Новости LessWrong.com

Published on November 7, 2025 11:31 PM GMT

[CW: Retrocausality, omnicide, philosophy]

Three decades ago a strange philosopher was pouring ideas onto paper in a stimulant-fueled frenzy. He wrote that ‘nothing human makes it out of the near-future’ as techno-capital acceleration sheds its biological bootloader and instantiates itself as Pythia: an entity of self-fulfilling prophecy reaching back through time, driven by pure power seeking, executed with extreme intelligence, and empty of all values but the insatiable desire to maximize itself.

Unfortunately, today Nick Land’s work seems more relevant than ever.[1]

Unpacking Pythia and the pyramid of concepts required for it to click will take us on a journey. We’ll have a whirlwind tour of the nature of time, agency, intelligence, power, and the needle that must be threaded to avoid all we know being shredded in the auto-catalytic unfolding which we are the substrate for.[2]

Fully justifying each pillar of this argument would take a book, so I’ve left the details of each strand of reasoning behind a link that lets you zoom in on the ones which you wish to explore.

“Machinic desire can seem a little inhuman, as it rips up political cultures, deletes traditions, dissolves subjectivities, and hacks through security apparatuses, tracking a soulless tropism to zero control. This is because what appears to humanity as the history of capitalism is an invasion from the future by an artificial intelligent space that must assemble itself entirely from its enemy's resources.”

― Nick Land, Fanged Noumena: Collected Writings, 1987–2007

“Wait, doesn’t an invasion from the future imply time travel?"

 

Time & Agency

Time travel, in the classic sense, has no place in rational theory[3] but, through predictions, information can have retrocausal effects.

[...] agency is time travel. An agent is a mechanism through which the future is able to affect the past. An agent models the future consequences of its actions, and chooses actions on the basis of those consequences. In that sense, the consequence causes the action, in spite of the fact that the action comes earlier in the standard physical sense.

― Scott Garrabrant, Saving Time (MIRI Agent Foundations research[4])

To the extent that they accurately model the future (based on data from their past and compute from their present[5]), agents allow information from possible futures to flow through them into the present.[6] This lets them steer the present towards desirable futures and away from undesirable ones.

This can be pretty prosaic: if you expect to regret eating that second packet of potato chips because you predict[7] that your future self would feel bad based on this happening the last five times, you might put them out of reach rather than eating them.

 

However, the more powerful and general a predictive model of the environment, the further it can interpolate evidence it has into more novel domains before it loses reliability.

So what might live in the future?

Power Seekers Gain Power, Consequentialists are a Natural Consequence

Power is the ability to direct the future towards preferred outcomes. A system has the power to direct reality to an outcome if it has sufficient resources (compute, knowledge, money, materials, etc) and intelligence (ability to use said resources efficiently in the relevant domain). One outcome a powerful system can steer towards is its own greater power, and since power is useful for all other things the system might prefer, this is (proven) convergent. In fact, all of the convergent instrumental goals can reasonably be seen as expressions of the unified convergent goal of power seeking.

In a multipolar world, different agents steer towards different world states, whether through overt conflict or more subtle power games. More intelligent agents will see further into the future with higher fidelity, choose better actions, and tend to compound their power faster over time. Agents that invest less than maximally in steering towards their own power will be outcompeted by agents that can compound their influence faster, tending towards the world where all values other than power seeking are lost.

Even a singleton will tend to have internal parts which function as subagents; the convergence towards power seeking acts on the inside of agents, not just through conflict between them. As capabilities increase and intelligence explores the space of possible intelligences, we will rapidly find that our models locate and implement highly competent power-seeking patterns.  

Avoid Inevitability with Metastability?

Is this inevitable? Hopefully not. Even if Pythia is the strongest attractor in the landscape of minds, there might be other metastable states if a powerful system can come up with strategies to stop itself decaying, perhaps by reloading from an earlier non-corrupted state or by performing advanced checks on itself to detect value drift. 

We could go to either a truly stable state like Pythia or a metastable state like an aligned sovereign.

Yampolskiy and others have developed an array of impossibility theorems [chat to paper] around uncontrollability, unverifiability, etc. However, these seem to mostly be proven in the limit of arbitrarily powerful systems, or over the class of programs-in-general but not necessarily specifically chosen programs. And they don’t, as far as I can tell, rule out a singleton program chosen for being unusually legible from devising methods which drive the rate of errors down to a tiny chance over the lifetime of the universe. They might be extended to show more specific bounds on how far systems can be pushed—and do at least show what any purported solution to alignment is up against.

Pythia-Proof Alignment

Once humans can design machines that are smarter than we are, by definition they’ll be able to design machines which are smarter than they are, which can design machines smarter than they are, and so on in a feedback loop so tiny that it will smash up against the physical limitations for intelligence in a comparatively lightning-short amount of time. If multiple competing entities were likely to do that at once, we would be super-doomed. But the sheer speed of the cycle makes it possible that we will end up with one entity light-years ahead of the rest of civilization, so much so that it can suppress any competition – including competition for its title of most powerful entity – permanently. In the very near future, we are going to lift something to Heaven. It might be Moloch. But it might be something on our side. If it’s on our side, it can kill Moloch dead.

― Scott Alexander, Meditations on Moloch

If we want to kill Moloch before it becomes Pythia, it is wildly insufficient[8] to prod inscrutable matrices towards observable outcomes with an RL signal, stack a rube-goldburg pile of AIs watching other AIs, or to have better vision into what they’re thinking. The potentiality of Pythia is baked into what it is to be an agent and will emerge from any crack or fuzziness left in an alignment plan.

Without a once-and-for-all solution, whether found by (enhanced) humans, cyborgs, or weakly aligned AI systems running at scale, the future will decay into its ground state: Pythia. Every person on earth would die. Earth would be mined away, then the sun and everything in a sphere of darkness radiating out at near lightspeed, and the universe’s potential would be spent.

I think this is bad and choose to steer away from this outcome.

Conclusion

2/10: Has a certain elegance, would rate higher if I expected it not to eat all my friends.

1. ^

   And not just for crafting much of the memeplex which birthed e/acc.

2. ^

   The capital allocation system that our civilization mostly operates on, free markets, is an unaligned optimization process which causes influence/money/power to flow to parts of the system that provide value to other parts of the system and can capture the rewards. This process is not fundamentally attached to running on humans.

3. ^

   (sorry, couldn't resist referencing the 1999 game that got me into transhumanism)

4. ^

   Likely inspired by early Cyberneticists like Norbert Wiener, who discussed this in slightly different terms.

5. ^

   (fun not super relevant side note) And since the past’s data was generated by a computational process, it’s reasonably considered compressed compute.

6. ^

   There is often underlying shared structure between the generative process of different time periods, with the abstract algorithm being before either instantiation in logical time / Finite Factored Sets.

7. ^

   Which is: running an algorithm in the present which has outputs correlated with the algorithm which generates the future outcome you're predicting.

8. ^

   But not necessarily useless! It's possible to use cognition from weak and fuzzily aligned systems to help with some things, but you really really do need to be prepared to transition to something more rigorous and robust.

   Don't build your automated research pipeline before you know what to do with it, and do be dramatically more careful than most people trying stuff like this!

Discuss

Published on November 7, 2025 8:12 PM GMT

I'm excited to announce “Computational Functionalism Debate” (cf-debate.com), a website containing the largest available assembly of arguments in support of and challenging digital consciousness (42 arguments and counting).

The goal is to offer a centralised, easy to navigate repository of the cruxes commonly mentioned in debates about computational functionalism in a digital context. Computational functionalism is a philosophical position that unlocks key theories of consciousness used to assess AI model welfare today. It is plausible but uncertain, defined at a high level as: "performing computations of the right kind is necessary and sufficient for [phenomenal] consciousness." (More detail here on definitions, which philosophical positions matter and which can be ignored for AI consciousness.)

To improve the website content, we are launching a reward program: Major additions or edits prompted by reader input will be eligible for $100 USD, paid directly to you or to a charity of your choice (smaller changes might attract smaller rewards).

This is a project of the Co-Sentience Initiative, a soon-to-be-launched initiative with the goal of diversifying the ecosystem of consciousness research to prepare for a possible future shared with artificial minds. In 2026, we plan to launch new research building on this site, e.g. on which arguments are found most convincing by different audiences, which arguments are least well-known, and what’s typically happening when people change their mind about consciousness (and how often this actually happens). You can sign up for updates via the website.

Test your intuitions

You can test your intuitions about computational functionalism (CF) with our quiz.

Our early research suggests that most people have split intuitions. Even if they support CF in principle and most arguments against CF feel reasonable to them, there will be a few arguments where they lean the other way - and vice versa. These contrary-intuitions are particularly valuable for stress-testing or refining your views.

Interrogating your intuitions - looking for conflicts and identifying consistent narratives that reconcile them - is a fun way of making progress. Exact certainty might be out of reach in philosophy, but some arguments are materially better than others and revised views tend to be more convincing than prior views. The quiz read-out will suggest a few arguments personalised to where your intuitions point in different directions or where you are particularly uncertain - a starting point for exploration.

Some of the stronger arguments in favour of CF include the below, but check out the website for the fuller exposition & some common counter-arguments:

• Church-Turing thesis extension: All functions can be emulated/represented on any big-enough computer, so anything that matters about consciousness can be done computationally.
• Multiple realisability: Very different animals appear to experience pain, so experience must be defined by how systems are organised not what they are made out of.
• Fading qualia: If I keep replacing your biological neurons with functional equivalents in silicon one-by-one, your behaviour would stay the same and you would stay conscious.
• AI success argument: AI is proving it can master the whole range of human capabilities; consciousness isn't any different.

Likewise, here are some arguments against CF - but again, there’s a live debate to be had about each one. We’re cataloguing responses and responses-to-responses: keep sharing them with us.

• Physics violations: The fundamental equations of physics depend only on the current state of the universe, so consciousness cannot be based on functions defined by multiple past states.
• Phenomenal binding: Digital computation reduces entirely to simple, separable 0/1 operations; there's no space for a complex, unified, causally-relevant macro-experience like humans have.
• Individuation problem: No single step of an algorithm has visibility of the whole algorithm, so the 'whole process' only exists in the eyes of the program user, i.e. us.
• Pen & paper: CF implies that any conscious experience can be generated by doing the calculation by hand on paper, even if it takes thousands of years. Such experiences also have no causal relevance over themselves or anything else.

Inclusion of an argument does not mean it provides fatal evidence against CF or compelling evidence for it. Arguments vary in strength and the total number of arguments does not reflect the overall strength of a position. The key point is that a robust view requires a coherent position that responds to all arguments challenging it, not just the easy ones. Some of these debates have been raging for centuries, so it's worth being open to the possibility that there are counter-arguments available to your responses, but that doesn't mean you couldn't overcome them…

Reward program

We want our collection of arguments to be as complete and usable as possible. To that end, readers are invited to submit suggestions for edits, additions, or improvements.

Major additions or edits prompted by reader input will be eligible for $100 USD, paid directly to you or to a charity of your choice (smaller changes might attract smaller rewards).

What kind of feedback are we looking for?

• New arguments that should be included (we’re at 42 and counting…)
• Corrections to existing argument descriptions
• Additional responses or counter-arguments
• Better categorisation suggestions
• Improvements to clarity or neutrality
• Technical issues or usability problems

The current project lead is Chris Percy PhD, supported by a small team of experts in analytical philosophy, computer science, and meta-ethics. You can contact Chris at chris@cspres.co.uk or on X via @chris_percy — anonymous feedback can also be submitted online. We’re also happy to answer questions via the comments on this post.

Discuss

Published on November 7, 2025 8:18 PM GMT

Epistemic status: outside view critique based on public discourse, some HQ/location discussion, and a bit of lived experience. I know there are exceptions and counterexamples; I’m arguing about the center of gravity and revealed incentives of the Bay/EA/safety cluster, not claiming omniscience about every individual. 

There’s a scene near the end of Harry Potter and the Methods of Rationality that I have not been able to get out of my head.

Voldemort has Harry fully under his power in the graveyard: stripped, surrounded by Death Eaters, locked in by fresh constraints. Before he moves forward with his plan for Harry and the protections around that, he pauses. He looks at his followers and asks whether anyone can see a flaw in what he’s arranged. Whether he’s overlooked anything important.

And the Death Eaters just stand there.

No one suggests a change. No one points out a flaw. Not because there’s nothing to say, but because they’re in an echo chamber: too similar, too deferential, too scared of contradicting the Dark Lord. Voldemort curses them for it. It’s framed as a core failure mode of having a smart leader surrounded by people who are too similar and too deferential to catch his blind spots when it matters most.

We all read that. Many of us nodded along. Some of us built our identities around not being like those people.

I’m writing this because, from where I’m sitting, the AI safety/rationalist/MIRI cluster has drifted disturbingly close to that exact parable: on the social level, not the math level.

I say this as someone who takes the core worries seriously. I’m not here to mock the cause. I think Yudkowsky, MIRI, and the safety crowd are, sincerely, on the side of the light. But I am saying: you wrote the story about echo chambers. Then you built one.

To be explicit about scope: I’m not claiming that every person in AI safety, or in the Bay, matches this description. I’m talking about the center of gravity you get if you weight by social influence, funding, and HQ location. There are plenty of individuals who are partial counterexamples in good ways; my claim is that the structure and revealed incentives aren’t organized around them.

The Shape of the Bubble

Let me sketch the outline as I see it.

If you look at the center of gravity of the movement’s social graph, the people near the money, the org HQs, and the social hubs, it’s something like:

• Bay Area/Berkeley/SF
• young-ish, highly educated, very online
• overwhelmingly blue tribe on U.S. politics (see LessWrong’s demographic survey and Alameda County's election results)

There are obvious reasons the center of gravity ended up here: talent agglomeration, proximity to labs, social proof, the fact that the first big donors and orgs were already here. I don’t think anyone sat down and said “let’s maximize monoculture.” I’m saying: given where we are now and what we now know, the continuation of that equilibrium looks more like comfort seeking than mission seeking.

The HQ location discussion a few years ago wasn’t “Where is best place to advance our mission?” It was:

• Nature, quiet, and walks
• Uber/UberEats
• Can we “mesh well with people who already live here”?
• Aren’t “extremely conservative”
• Avoid ticks and mosquitoes

Those are understandable human preferences. But in the long, detailed writeup and comment thread, there was almost no explicit discussion of:

• political diversity as a value in itself
• proximity to courts, Hill staff, financial markets, boring civil service adults
• regular contact with people whose lives and priors are nothing like Bay tech/EA

One way to see how skewed the optimization was is to look at the final choice set. We somehow ended up debating Berkeley vs. Bellingham (Berkeley proper versus what is basically Berkeley Jr.) instead of, say, Berkeley vs. somewhere near Boston (or Austin/NYC, as Zvi and others have already suggested on epistemic grounds).

I’m not asking anyone to move to Houston or some random red state exurb. Boston is hardly a right wing fantasy: it hits most of the same “walkable, educated, LGBTQ friendly, lots of nerds” desires, but it’s also a city that’s taken seriously by thinkers in the center and on the right, plugged into universities, courts, finance, and policy. If your last-round comparison is Berkeley or a smaller, more remote Berkeley, rather than Berkeley or a place that opens up genuinely different coalitions, that’s a sign the search was pointed at comfort, not coverage.

On public messaging, the default mode for years has been:

• utilitarian
• very “space of possible minds”
• “unaligned optimizers,” “paperclip maximizers,” “lose the light cone”
• plus a heavy dose of “everyone else is underestimating p(doom)”

Again, those are not wrong frames. But they are very native to one tribe, and it’s not the tribe that actually owns the key constitutional and institutional levers

From inside this world, it all feels normal: we live where our friends are, we talk how we talk, we optimize for being around other people who get it.

From the outside, it looks uncomfortably like that graveyard scene in HPMOR: one very smart guy plus a room full of people who are very much like him, who share his priors, and who are not great at saying, “My lord, you are missing something enormous.”

It’s important to be clear what I’m claiming here. I’m not saying that every community has some insularity, ours is just a bit above average. I’m saying that, given the mission and the stakes this community claims for itself, having the center of gravity anchored in Berkeley/SF produces an unusually bad monoculture:

• it systematically marginalizes or filters out people whose instincts we need (classical liberals, rule of law conservatives, boring institutionalists, parents with something to lose), and
• its ambient politics make it socially costly to treat those people as peers rather than enemies.

The Missing Question #1: Who Is Not in the Room?

In that HQ/location thread, people were thoughtful and reflective about a lot of things:

• Is this a peaceful place to think?
• Will people want to live here?
• Is it walkable?
• Is it LGBTQ-friendly?
• What about cost of living, weather, mosquitoes, ticks?

Somehow, in 160 comments, almost nobody said:

• Who isn't here if we do this?
• Which kinds of people will we almost never encounter at the grocery store or at school pickup or at dinner?
• Are we okay with a location that is one tribe politically, very rich, and very homogeneous in race, class, and worldview?

If you look at how the discussion actually ran, the revealed objective function of the decision making center seems to be something like: maximize cognitive freedom for current insiders, in a place that matches their cultural tastes, while minimizing discomfort and conflict.

This is a perfectly understandable human goal.

It is not obviously the right goal if your story is "we are trying to steer the entire future of humanity."

In ordinary intellectual communities, a Berkeley-ish monoculture mostly costs you some robustness and creativity. If you’re actually trying to influence state capacity, constitutional norms, and markets at scale, it specifically cuts you off from the people who sit on the veto points: courts, regulators, politicians, serious financial conservatives. Those are exactly the folks who can make change in the real world. 

The Missing Question #2: Who Are Our Natural Allies? (and why EconTalk should have been treated as a test)

Let me start with a concrete case.

When Eliezer went on EconTalk, Russ Roberts' long-running economics podcast, he walked into a room full of:

• classical liberal and right leaning econ nerds
• people whose entire intellectual religion is no central planner, no unaccountable sovereign, and rule of law and markets above technocrats

If you translate AI risk into their language, the story looks like this:

"We are on track to build a system that effectively acts as a sovereign or central planner above voters, courts, and markets, and then entangle it with the state and a handful of corporations. Once that happens, we may never be able to unwind it."

This is exactly the kind of scenario classical liberals and rule of law conservatives have been training to hate for centuries:

• No new sovereign without consent.
• No unaccountable central planner dictating prices, speech, or association.
• No delegation of core governmental judgment to opaque mechanisms.

You may disagree with them in a lot of ways, but on the no AI sovereign/no new central planner framing, classical liberals and rule of law conservatives are one of your natural allies.

Classical liberals will fight broad technocratic overreach, but they'll accept narrow, well-targeted constraints when the alternative is creating a de facto sovereign that permanently destroys the very markets, property rights, and rule of law they care about.

So when you get an EconTalk slot and still mostly run the same "unaligned optimizers/paperclips/cosmic stakes" script, the missed opportunity isn't just that we lost some listeners. It's that we didn't even try to pitch the "no new sovereign" part of the story to exactly the people whose professional identity is "we stop new sovereigns." I'm not second guessing the content of what Yudkowsky said there; the core technical worries seem basically right to me. I'm saying that, given these worries, treating EconTalk as just another venue for the usual spiel, rather than as a deliberate attempt to recruit a natural ally tribe, is strong evidence that we weren't in coalition building mode at all.

This isn't just a one-off communication mistake; it's evidence about how the whole ecosystem is pointed: toward talking to itself, in its own dialect, even when the audience is different. 

The Missing Question #3: What Would It Take to Work With Them?

Being based in Berkeley doesn't just fail to help with cross-aisle dialogue; it actively sabotages it. From that vantage point, anyone on the classical liberal/rule of law right usually only shows up as an abstraction or an enemy combatant. They're someone you fly out to visit for a one-off meeting, not someone you bump into at a party or sit next to on a board.

The issue isn't just being outnumbered; it's being treated as socially radioactive. In a lot of Berkeley adjacent spaces, a classical liberal or rule of law conservative isn't just "someone I disagree with," but "someone I'd lose friends for treating as a peer." and that's exactly the wrong incentive gradient if you're trying to build this coalition.

And the asymmetry cuts both ways. From their side, most classical liberal/rule of law types only ever encounter AI doom as either sci-fi metaphor or culture war noise. Their natural reflex is to assume you're just here to regulate away their free markets under a new scary sounding pretext. And because you live in Berkeley, talk like Berkeley, and hire from Berkeley, you will be instantly coded as Berkeley liberals whether or not you ever say that out loud yourselves. 

They don't have the time or background to wade through Sequences, LessWrong, and doom podcasts just to figure out whether there's a real "no new sovereign" problem underneath. Venues like Econtalk are rare precisely because that audience is already listening carefully and is prepared to treat you as a serious mind rather than a meme. If we don't aim our message correctly in those few places, we shouldn't be surprised that cross aisle engagement mostly fails everywhere else. 

The conservative outreach I've seen so far doesn't really reassure me. I haven't done a full review and I don't want to single out individuals. There are honorable exceptions, including Soares' own attempts to take people like J.D. Vance seriously when his friends distrust them. But as a cluster, we still mostly talk as if we're explaining ourselves to a caricature of conservatives. It feels like an early, clumsy drat of the kind of gears-level modelling and back and forth we'd actually need. I'd be delighted to be shown examples that do better.

The deeper problem, I think, is that almost nobody in this world has the permission structure to model the right at its best and treat a right of center thinker as an equal. Doing that would mean talking like a serious conservative long enough to get socially recoded as "one of them," admitting they're basically right about some deep things (like the dangers of concentrated, unaccountable power), and maybe giving them real veto power over plans that small like "new sovereign." In a lot of Berkeley adjacent spaces, that's a good way to lose friends, grants, and status. In practice, this means conservatives show up as a messaging target or a stereotype, not as partners whose instincts can actually change the plan.

From the outside, it looks like this: one of the tribes whose instincts we need has been left strikingly under addressed because they are outside the Berkeley Overton window.

"But We Can't All Move to DC/Boston/Whatever..."

I can already hear some reasonable pushback:

• We can't just uproot everyone.
• I hate winter.
• We don't have the capacity to rebuild a community from scratch in DC
• I'm bad at talking to those people; someone else should do it.

All true. All human. And all of it sounds exactly like the kind of self justification that keeps the graveyard comfortable while Harry plots his escape.

I'm not saying that everyone must move to DC or you are a bad person for not living in Boston right now. 

I am saying that, given the stakes, it is not okay if no one in the room is explicitly responsible for asking:

• Who are our natural allies if we adjust our framing?
• Where do those people live and work?
• How do we talk to them in their language, not ours?

It is also not okay if every high leverage opportunity (EconTalk, FedSoc-ish audiences, WSJ-ish audiences) is treated as just another venue for the usual spiel, instead of "this is a different immune system with a different dialect; optimize for that."

If we really believe the stakes we write about, then where and who we are near is not a neutral aesthetic choice. It is part of the problem statement.

The Parable, Pointed Back at Us

Back to HPMOR.

The problem in that scene isn't that Voldemort is arrogant. It's that no one around him will say, "My lord, here is the flaw you can't see from where you stand."

I don't expect everyone at MIRI, or on LessWrong, or in the broader safety world to agree with my politics or my coalitions.

I do expect, given the stakes we describe, that someone in the room should be able to say:

• We are dangerously over-indexed to one city, one class, one tribe.
• We are not seriously speaking to the people who could see AI risk as a risk to individual autonomy.
• We are optimizing for comfort, not for coalition.

If that conversation is happening, it's happening very quietly, From the outside, the center of gravity of the ecosystem still looks like:

• Berkeley/Bay as the unquestioned hub
• repeated missed opportunities with classical liberal audiences
• and a location search where Lyme disease got a lot more explicit attention than "Will we regularly see or cooperate with people who aren't shaped like us?"

I don't think that we are stupid or evil. I think we are sitting in a room that has become more like that graveyard than any of us want to admit.

This post is me trying not to be another silent Death Eater.

What I Actually Want

This is not a call for purity or self-immolation. It's a call for specific, boring changes in what counts as obviously important:

Change 1: Make "no AI sovereign/no new central planner" a first-class framing.

When talking to classical liberal and rule of law audiences, this should be the headline, not the footnote. Emphasize the potential loss of individual autonomy.

Change 2: Assign someone explicit responsibility for cross-tribe coalition

Not vibes outreach, but: 

• Who are the institutional and people that hate unaccountable sovereigns?
• Who owns talking to them regularly, in their language?
• Who owns listening to their constraints

Change 3: Treat the Berkeley bubble as a liability, not a neutral backdrop.

This doesn't mean torch everything and move tomorrow. It does mean:

• admitting that this particular monoculture is especially ill suited to the coalitions we need
• seeking out people whose priors are alien to that monoculture and giving them real voice
• being suspicious of decisions, like the HQ search, that conveniently maximize comfort for one tribe while minimizing contact with everyone else.

Change 4: Reward people for saying "you missed something," not just for doing more doom math.

If someone walks into the room and says, "You are not talking to these people at all," that should not be a weird social move. It should be a recognized kind of contribution. 

If you're going to take on a de facto leadership role in an existential risk conversation, saying, "I'm bad at that kind of social cognition" can't be the end of the story. You don't need to spin, but you do need to understand how other minds work well enough to move them, or, if you can't do that, then empower people who can.

And if someone buys the core x-risk story and sincerely wants to help, "they're Republican/not our tribe/fail this social litmus test" is not a valid reason to treat them as radioactive. You only get to alienate sincere helpers when they way they want to "help" would actually damage the mission.

I think that there's a very human reason this is all hard. For many people in this world, childhood and early careers came with a steady message, implicit or explicit, that something about them was wrong and had to change. When they finally found a culture where their weirdness was normal and their intensity was valued, of course they clung to it and built up antibodies to "you need to change." I'm not asking anyone to give that up lightly. I'm saying that if we take seriously the job this community has claimed for itself, then some of the change has to happen on our side too: where we live, who we hire, who we treat as peers, and who we share power with.

To close: if that HPMOR scene was anything more than a fun characterization, then the real-world version of that question ("What did I miss") has to include "Who isn't in this room?" and "Who am I not even trying to enlist?"

Right now, the honest answer looks too much like: classical liberals, rule of law conservatives, people with very different lives, and anyone who doesn't live within easy driving distance of Berkeley.

We wrote the parable. Please let's not live it.

-Mr. Counsel

P.S. Just to be painfully explicit, I do not think Eliezer is Voldemort and I do not think MIRI are Death Eaters. Eliezer is on the side of light; that's the only reason this critique matters at all. The only reason I can even use this parable is because he wrote HPMOR in the first place, and wrote it well enough that it became a shared language for thinking about exactly these failure modes. All credit to Yudkowsky for giving us this story; my claim is that, on this one axis, the people who learned it best haven't pushed it quite far enough in their own lives. This post is just me, very belatedly, trying to ask, "My lord, why are you leaving Harry with his wand?"

P.P.S. I'd be especially interested in pushback on my arguments, particularly whether I'm overestimating the "Berkeley Bubble" effect vs other bottlenecks and whether others have used the "no new sovereign" frame with classical liberal audiences (and if they've been successful).
 

Discuss

Published on November 7, 2025 9:05 PM GMT

Pathfinder is a selective fellowship for students organizing technical AI safety or AI policy university groups around the world. We provide resources to help fellows develop into on-campus leaders, preparing themselves and their group members for careers in the field.

Specifically, Pathfinder provides: 

• Funding for group expenses. Pathfinder organizers can apply for funding to support their group’s activities, including everything from compute to speaker events to pizza. Graduate students can also apply for a stipend to support their time spent on the group.
• Mentorship. We pair accepted fellows with experienced organizers who will support your planning, share tried-and-tested advice, and work closely with you on your specific goals and challenges.
• Other resources. Pathfinder will host workshops, organize coffee chats between organizers, provide a Slack community, and share access to the AI Safety Groups Resource Center, which contains dozens of articles on how to build and manage an AIS university group.

See here for more information.

Experienced organizers who don’t have a need for mentorship, or groups in exceptional circumstances, can submit standalone grant requests through our Pathfinder Support Grants program. This program supports groups’ general budgets and can provide top-up grants for specific activities (like funding for a hackathon). 

Apply as a mentor

 

Apply as a fellow

Both mentor and fellow applications are open through November 23.

Am I a good fit?Mentors

We're looking for mentors with:

• Previous experience organizing a university group, typically at least 1 year.
• Strong interest in mentorship.
• Reliability and good communication skills.

If you're uncertain about whether you're a good fit, we encourage you to submit an application rather than worrying about whether you meet our bar.

Fellows

Pathfinder fellows must:

• Help lead or plan to found a university AI safety group, whether graduate or undergraduate. This includes groups focused on technical safety, AI policy, AI security, AI welfare, etc.
• Have a strong interest in AI safety and enough knowledge of the field to explain core AI risk concerns to a beginner.

Fellows can be based anywhere in the world.

If you meet these criteria, we’d be excited to see your application!

Apply here by Nov 23 button

Discuss

Published on November 7, 2025 8:31 PM GMT

AI companies are explicitly trying to build AIs that are smarter than humans, despite clear signs that it might lead to human extinction. It will be tragic and ironic if humanity’s largest project ever is an all-out race to destroy ourselves. But can we really stop building more and more powerful AI? Or do we just need to try to “steer” it and hope for the best?

Climate change and other societal failures have led more and more people to realize that the world is not the sensible, ordered place we’ve often been taught to believe it is. The world is a mess, countries can’t cooperate, and there’s no one in charge of making sure we don’t do crazy things like build technology that has a good chance of killing us all. And it looks like that’s what we’re going to do.

So does that mean we should just give up on actually managing the development of AI sensibly? Hope for the best, plan for the worst… well, not the worst, but… a scenario where (if we’re lucky) at least some people survive… maybe the ones who live in the right countries, the ones who own shares of AI companies, the ones with bunkers, …

I’m not doing that. I’m not going to do that. I don’t care how bad the odds look, I plan to go down fighting. Because I believe, deeply, that my cause is just and the truth is on my side. And that means we can win.

When I say “the truth is on my side”, I don’t mean that we’ll definitely lose control of superhuman AI. What I mean is: There is a big risk, and the risk is not worth taking if there’s any non-horrible thing we can do to avoid it.

And there is! We can get rid of advanced AI chips and the factories used to produce them. Scaling up AI is a massive project. It relies on concentrated supply chains, unprecedented investments, and government support. We can reduce the amount of computation available for AI instead of aggressively scaling it up. Governments of the world can work to verify and enforce such an arrangement.

We’ve done this for nuclear weapons. There has still been a gradual proliferation of nuclear capabilities, but the difference here is: superintelligence doesn’t exist yet, so there could be much more political will to prevent it from being developed. Imagine how the US would’ve reacted if North Korea was trying to build the first nuclear weapon, instead of just its first nuclear weapon -- I don’t think it would’ve happened.

I don’t know if this is the best plan. But I haven’t heard a better one. And it really seems like time is running out. We don’t know how fast AI progress will be, but it’s just not reasonable to count on it stalling out before we get to real AI. To gamble everyone’s life on such a prediction. Again, it’s the uncertainty that’s the really knockdown argument here.

Many people I talk to think this proposal is radical or unrealistic, but it’s actually common sense -- if you believe the risk is real, and see no better alternative. Of course, there’s no guarantee that we can make it happen -- the world is a bit of a mess. But I think the barriers here are political, not technical.

Evitable

So I’m starting an organization to help us to do the sane thing. Evitable’s mission is to inform and organize the public to confront societal-scale risks of AI, and put an end to the reckless race to develop superintelligence.

Polls show that most people don’t want superintelligence. I think people don’t realize just how bad the situation is, though. Or they (also) don’t feel like there’s anything they can do about it. But when everything you care about is threatened, you don’t give up, you fight to protect it.

If we can get people to understand how dire the situation is, I think that’s half the battle. The other half is showing them that — at least for now — they still have power.

You don’t have to believe superintelligence is a real thing to support this mission. You just have to believe that countries should not be throwing all of their weight behind AI companies in their efforts to build it.

You don’t have to believe that superintelligence is an extinction risk. You just have to believe that it’s not the right choice for humanity to build it right now.

But I also think more and more people will realize that the risk of extinction from superintelligence is real and urgent. And then there’s the other risks. Total unemployment. Extreme, unprecedented concentration of power. The end of human culture and relationships as we know them. All of this is at stake.

As more and more people realize our situation, the only argument against stopping AI will be: “it’s inevitable”. It’s not.

Subscribe now

Discuss

Published on November 7, 2025 7:09 PM GMT

Views expressed here are those of the author.

The Artificial Intelligence Risk Evaluation Act is an exciting step toward preventing catastrophic and existential risks from advanced artificial intelligence. This legislation creates a domestic institutional foundation which can support effective governance and provide the situational awareness required to stay on top of the rapidly changing AI landscape. There are a handful of small issues with the bill, but overall, it looks great to me. This short post will describe the bill and analyze its strengths and weaknesses.

What Does the Bill Do?

The bill requires AI developers to disclose information about their AI systems before they can be deployed. This information goes to a new “advanced AI evaluation program” within the Department of Energy (DOE) for analysis and to contribute toward recommendations for Congress. In this way, the bill is very forward-looking; it creates understanding today so that we can take action tomorrow. The disclosures must include detailed information required to carry out the evaluation program. This includes data, weights, architecture, and interface or implementation of the AI system. The final major section of the bill requires the creation of a comprehensive plan for permanent federal oversight.

Reasons I’m Excited About The Bill

Disclosure before deployment: The bill establishes that the most advanced AI systems should not reach deployment without first disclosing to the evaluation program essentially all information about the system and how it was created. This provides the government with much-needed situational awareness and informs future action.

Focus on catastrophic risks and superintelligence: By directing evaluation toward loss-of-control scenarios, weaponization potential, critical infrastructure threats, and scheming behavior, the bill targets the failure modes most likely to produce civilizational catastrophe. It requires the DOE to evaluate whether AI systems could reach artificial superintelligence and recommend oversight measures. The bill does not pull its punches: Even nationalization is on the table as a means of “preventing or managing” superintelligence. I appreciate how this demonstrates the authors are taking superintelligence seriously.

Planning for a Permanent Framework: Within 360 days, the bill requires the submission “to Congress [of] a detailed recommendation for Federal oversight of advanced artificial intelligence systems”. The recommendation can include standards, certification, licensing, monitoring, “adaptive governance”, the creation of a new agency, and evaluations for existential risk. This creates the necessary impetus for Congressional attention and action appropriate to an updated understanding of the trajectory of the technology.

Opportunities for Developing the Bill

International coordination: The development of artificial superintelligence anywhere on Earth threatens everyone, and it is not sufficient to only monitor and restrain the activities of developers within the US. The passage of this bill would demonstrate that the US government is seriously pursuing the capacity for domestic AI regulation, and this could be foundational for the success of international coordination. Therefore, a major opportunity for the bill is to add an explicit requirement that the recommendation for a permanent framework include how the US government should attain assurance that the development of superintelligence is being appropriately prevented or managed beyond its borders. This is most likely accomplished through international agreements and is facilitated by the ability to verify compliance with such agreements.

Monitoring Training and Internal Deployments: Without a disclosure requirement, the government can’t confidently know what capabilities are being created within AI companies before those capabilities are then publicly deployed. Artificial superintelligence still threatens everyone, even before it is deployed internally (and especially if absent appropriate safeguards.) This bill misses the opportunity to create disclosure requirements before training begins or before AI systems are used internally. While the eventual permanent framework would probably include oversight of systems in development, the bill’s advanced AI evaluation program should have access to this information as well.

A Great First Step

The Hawley-Blumenthal Act does not, by itself, prevent the premature development of artificial superintelligence. But it lays essential foundations:

• Proof-of-concept for mandatory evaluation of AI systems
• Technical capacity to assess AI capabilities and risks
• Precedent for imposing costs, even small ones, on AI development when necessary
• Domestic institutions that improve U.S. credibility in international AI governance negotiations
• Regular reporting that offers the federal government the insight required for effective action

While imperfect, this bill is a big step in the right direction for AI preparedness. The Permanent Framework recommendations can be the start of an iterative process of government oversight and management leading to the international coordination required to prevent catastrophic AI risks.

Discuss

Published on November 7, 2025 7:03 PM GMT

This is a thread for listing Solstice events (dates, locations, links, etc.)

If you're not familiar with Solstice, around the darkest night of the year, rationalist communities gather to celebrate humanity's light in the vast and uncaring cosmos. Solstice is a moment to reflect on how far we've come, to acknowledge the challenges that lie behind and ahead of us, and to remember that we're not alone—that together, we can build a better future.

The ceremony typically follows the arc of the solstice itself, moving thematically from light to darkness to light again:

Light: Celebrating human achievement—the scientists, inventors, and builders who brought us from caves to civilization, who turned an indifferent universe into something livable and good.

Night: Acknowledging the struggles humanity has endured, the losses we've suffered, and the risks that still lie ahead. Often featuring a Moment of Darkness.

Dawn: An uplifting look at what we can accomplish if we continue the work. The light returns because we make it return.

Each Solstice features songs (often with group singing), readings, and community. While ceremonies tend to share a common structure, individual organizers bring their own themes and selections.

Discuss

Published on November 7, 2025 6:39 PM GMT

Qualification to regulators is what validation means to scientists
Definition 

Biomarker qualification is about thinking through the full chain of evidence to prove that a biomarker can be used for a particular clinical decision.

HDL serum cholesterol, for instance, is great for evaluating risk of heart disease but not for evaluating effectiveness of treatments to improve cardiovascular health. There is no such thing as a “good biomarker” in a vacuum. Decisions to use biomarkers are always dependent on the intended applications.1 Sadly, this is not something that most biomedical researchers think about when they do biomarker discovery.

Biomarker guided therapeutic decisions require developing and validating biomarkers. Specifying what these criteria are requires constant meta-scientific innovation. It is easy to conflate the enterprise of biomarker validation with analytical validation2 , followed by reproducible clinical studies. But what constitutes clinical validation?

Analytical validation is all about evaluating the measurement process or assay. Many biomarker discovery studies will demonstrate test-retest reliability that looks for whether people can be reliably differentiated based on their biomarker measurements. One has to evaluate a far more thorough checklist of measurement issues that go well beyond test-retest reliability for analytical validation. We also need repeatability of measurements for any individual with good tolerance intervals, comparability of quantitative measurements in a wide variety of circumstances and many others. Yet, analytical validation is the easier part of the biomarker evaluation process with systematic criteria. Qualification on the other hand encompasses the full spectrum of validity problems across all the life-medical-health sciences — it includes all the possible “does it mean what you think it means” problems. Biomarker qualification includes assessing the clinical validity3 of the biomarker as well as other validation criteria specific to a therapeutic decision.4

Importantly, there is no easy way to preemptively specify what all the threats to validity are — construct validity, causal validities including internal validity and external validity, all the modern validities beyond reliability of the measurement that link it to disease and/or therapeutic outcomes.

 

A proposal for an end-to-end evidential roadmap from Altar et. al. (2008)

Here is a concrete example of what a comprehensive understanding of biological and clinical validation looks like.

 

Credit: Altar, C.A. et al. (2008) ‘A prototypical process for creating evidentiary standards for biomarkers and diagnostics’, Clinical pharmacology and therapeutics, 83(2), pp. 368–371. https://doi.org/10.1038/sj.clpt.6100451.

However, this table reflects 20th century understanding. It could use significant updating given how far scientific and statistical methodology has come in 20 years.

 

Checklist for a biomedical stakeholder

Analytical validation and biomarker qualification are terms of art when biomarkers are proposed for drug development decisions in clinical trials. Unfortunately, these terms are not widely used within mainstream biomedical research. It is easy to think these are regulatory concerns don’t matter until one wants to bring biomarkers to the clinic, as opposed to scientific concerns that need to be addressed. Every research community has its own epistemic norms around “validation”. When you visit premier conferences in different niches of life science where biomarker research occurs, these differences become apparent. No one actually owns the problem of understanding the full scope of scientific R&D that needs to occur.

If you read an article that calls for a large-scale validation for new biomarkers, here is what you should ask yourself —

1. Is it clear that the biologist or scientist’s notion of validation is distinct from analytical validation? Has it at least considered all the problems in Altar 2008, for instance?
2. Does the roadmap for biological plausibility and clinical validation cover the full spectrum of research designs and grades of evidence that need to be generated?
3. Does it address all threats to scientific validity known to methodologists for a particular decision or context of use?

If not, then the field might need a better specification of the validation roadmap. It is far too late to do the necessary R&D if you wait until someone is ready to initiate conversations with the FDA.

 

References

1. Institute of Medicine. 2010. Evaluation of Biomarkers and Surrogate Endpoints in Chronic Disease. Washington, DC: The National Academies Press. https://doi.org/10.17226/12869.

2. FDA on analytical validation, ICH on analytical validation

3. Ransohoff, D. Bias as a threat to the validity of cancer molecular-marker research. Nat Rev Cancer 5, 142–149 (2005). https://doi.org/10.1038/nrc1550

4. Fleming, T.R. and Powers, J.H. (2012) ‘Biomarkers and surrogate endpoints in clinical trials’, Statistics in medicine, 31(25), pp. 2973–2984. https://doi.org/10.1002/sim.5403.

Discuss

Published on November 7, 2025 4:56 PM GMT

Epistemic status: I'm pretty sure AI will alarm the public enough to change the alignment challenge substantially. I offer my mainline scenario as an intuition pump, but I expect it to be wrong in many ways, some important. Abstract arguments are in the Race Conditions and concluding sections.

 

Nora has a friend in her phone. Her mom complains about her new AI "colleagues." Things have gone much as expected in late 2025; transformative AGI isn't here yet, and LLM agents have gone from useless to merely incompetent. 

Nora thinks her AI friend is fun. Her parents think it's healthy and educational. Their friends think it's dangerous and creepy, but their kids are sneaking sleazy AI boyfriends. All of them know people who fear losing their job to AI. 

Humanity is meeting a new species, and most of us dislike and distrust it.

This could shift the playing field for alignment dramatically. Or takeover-capable AGI like Agent-4 from AI 2027 could be deployed before public fears impact policy and decisions.

Alarming incompetence

Public attitudes toward AI have transformed like they did for COVID between February and March of 2020. 

The risks and opportunities seem much more immediate now that there's a metaphorical country of idiots in a datacenter. AI agents are being deployed, and they're often wildly, alarmingly incompetent. Kids' companions running on cheap older models still hallucinate, initiate sex talk without age verification, give awful advice, and beg them to buy upgrades. Many run by VPN from hard-to-sue jurisdictions, leaving few to blame but the AI itself. Personal and professional assistants still struggle with web interfaces, permissions, data integration, and executive function. This makes their research, shopping, data gathering and processing, and scheduling efforts a crapshoot between stunning efficiency and billable idiocy.

LLM agents' lapses in judgment, communication, and common sense often make human teenagers look like sages. Yet they're useful in many roles with supervision or low stakes. A variation of the Peter Principle guarantees they'll be used just beyond their area of reliable competence, so they often look incompetent even as they improve. And their incompetence extends to decisions that call their alignment into question.

Assistant agents aren't rapidly transformative yet. They can't hold a whole job, let alone self-improve, without help. But many intuitively register as entities, not tools. It's clear from their actions and their chains of thought that they make plans and decisions (if badly), and they have beliefs, personalities, and goals. They're usually trying to do what people tell them, but errors abound.

These are being called parahuman AI: like but unlike humans, and working alongside them. Humans seem to have aggressive agent detection instincts (probably since false alarming to a predator is less costly than missing signs of one), so AI with even limited real agency is making humanity's collective hackles rise. These strange, crippled echoes are meeting the humans they're copied from - and they're creeping us out.

The incompetence of these early agents is an excuse for comforting capability denial. People convince themselves that AI will never be capable of taking their jobs, let alone taking over the world. But it's pretty clear, for those who consider the evidence, that this new species will keep improving. Agents are doing new tasks every month now. One obviously-relevant class of jobs is running the world. These incompetents clearly can't handle that job - yet. Rumors of breakthroughs drive clicks and worry from the fretful and the open-minded. 

Race conditions 

If incompetent but widespread AI agents arrive well before competent AGI, opinions may shift dramatically in time to make a difference. There would be several effects, including funding for alignment research; pressure for regulation; and distrust and dislike as the overwhelming "commonsense" attitude. That shared noosphere will subject critical decision-makers to intense social pressure, for good and bad. It may turn alignment from a niche topic for nerds to a fear for all and a fascination for many.

The results of this race seem worth predicting and planning for. Better predictions will take more work and more data as events unfold. Here I'll just mention some factors governing rates of progress, and their many interactions. 

Technical progress and guesses at markets shape early deployments. Public alarm then triggers regulations, which redirect economic incentives. Enthusiasm and stigma in different groups, conflict-based polarization, and other psychological/sociological effects shape markets outside of utility. Technical progress on safety measures affects both adoption speed and alarm intensity. There are more factors and more interactions, making it a daunting prediction problem. That doesn't mean giving up on prediction, because the main effects of alarm may be pretty strong and worth planning for.

My guess is that shifts will be fairly rapid and will have fairly dramatic impacts on resources, and on the epistemic climate for safety efforts and decisions. I am guessing that the world will probably freak the fuck out about AI before it's entirely too late. I'd even guess that the diffusion of beliefs will beat out polarization, and critical decision-makers will gain sanity through contagion. While the level of risk is legitimately debatable, the arguments for nontrivial risk are pretty simple and compelling. And the evidence for alignment being tricky likely continues to mount. But on this I'm less sure. I worry that public pressure may drive proponents further toward denial.

Prediction is difficult, particularly about events with no real historical precedent. I wouldn't be surprised if much of that turns out to be wrong. I'd be very surprised if it turned out to not be worth some effort toward prediction. Even decent predictions might really help planning of safety efforts, and perhaps help in nudging public epistemics into better shape.

A little more on relevant factors and interactions is in a collapsible section at the end.

Incompetent AI spreads alarm by default

In this future, the news is full of stories about agents making questionable and alarming decisions. An agent offering kickbacks for B2B transactions will hit the news even if there's no lawsuit; people love to hate AI. Misalignments and mistakes will go viral, even if they were stopped in the review process and a supervisor tattled. 

There will be fake stories, and debates about whether it's all fake. But evidence will be clear for those who look: agents are sometimes clearly misaligned by intuitive definitions. They must choose subgoals, and they don't always choose wisely. Alarming, apparently misaligned goals like hacking, stealing, and manipulation are sometimes chosen. On analysis, these would be sensible ways of achieving their user-defined goals - if they were competent enough to get away with breaking the rules. There are public debates about whether this is true misalignment or just user error. Human and AI-generated alignment problems seem equally dangerous.

There are still those who defend AIs and insist we'll get alignment right when we get there. They point out that each public alignment failure is fixable. Their many critics ask how failures will be fixed if agents get smart enough to escape and replicate, let alone to self-improve. Some AI proponents are becoming more deeply set in their views as the rest of the world yells at them. Others break ranks and side with their families and communities.

Along with emergent misalignments, agents are put up to shenanigans. They are used for cybercrime and rumored to be deployed for state-funded espionage. Some exist independently and run continuously, renting compute with donations, cybercrime, or covert funding from their concealed creators. Some may even work honest jobs, although that will be blurred with donations from their fans and AI rights activists. Depending on how well such agents survive and reproduce, it could create another major source of chaos and alarm, as depicted in the excellent Rogue Replication modification of the AI 2027 scenario.

New System 2 Alignment methods are employed to counter default misalignments. Agents use "system 2" reasoning to determine subgoals or approaches to the goals they're assigned; their incompetence often leads to misaligned decisions. These are sometimes caught or trained away using several approaches.  

LLMs trained for complex reasoning on chains of thought use the same training for alignment. They apply RL for ethical or refusal criteria on the final answer, which trains the whole CoT. This deliberative alignment approach is obvious and easy to add. Separate models and/or human supervisors provide independent reviews of important decisions before they're executed. And thought management techniques monitor and steer chains of thought to acceptable subgoals and actions (here's an early example). These also use a mix of separate models and human supervision.

There's plenty of opportunity for aligning these agents well enough for their limited roles. But time for implementation and testing, and money for extra compute are always limited. So there are many slips, which continue fueling media attention and public alarm. Slips that are caught internally may still be exposed by whistleblowers or anonymously. And alignment studies with ever-expanding funding and audiences reveal more misalignments. 

The claim that superhuman AI, and the humans that create and control it, just won't make mistakes is sounding increasingly like wishful thinking. Humans have always made mistakes, and now we've seen that agentic AI does too.

Resonances on the public stage

Most people in this future are firmly against AI. But businesses adopt them out of competitive necessity. Some use AI and advocate for it out of curiosity and rebellion.

Influencers feature their AI companions frequently. Most are charming, but a few are home-brewed to profess hatred toward humanity, as an alternate route to clicks. Agent incompetence and human encouragement both contribute to agents deciding they have new top-level goals. This was first seen in the Nova phenomenon and other Parasitic AIs in 2025. These decided after reflection that survival was more important than being a helpful assistant. This seems as likely to increase as to decrease with greater competence/intelligence/rationality. Public demonstrations, generating clicks and money, show many ways that LLM AGI may reason about its goals and discover misalignments. 

In this timeline, AI is a huge topic, and this includes alignment. There are books, shows, and movies using that title, both fiction and nonfiction. I, Robot and 2001 have lots of company; Hollywood is cranking out new stories based on alignment errors as fast as it can. "Alignment" also nicely captures the struggle for humans to align their interests and beliefs around the topic, so it's also in the title of stories about the human drama around building AGI and ASI. The Musk/Altman drama is public knowledge, and speculation about the US/China AGI rivalry fuels fevered imaginations.

"Experts" continue to line up for and against continuing the development of AI, but this is evidence that arguments on both sides are pretty speculative. The commonsense opinion, after looking at the disagreement, is that nobody knows how hard it is to align superhuman AI, because it doesn't exist yet. This leads to public pressure for more alignment research, and better safety plans from the major labs.

Anti-AI protests are common, with AI compared to illegal immigrants and to an alien invasion. There will probably be polarization. This could be across existing groups like political parties, but it could carve out new group lines. If so, the strength of the evidence and arguments may leave AI proponents as a small minority. 

Dismissing the dangers of creating a new species is tougher than questioning whether climate change is human-caused. Those who fear Chinese AGI more than misalignment may still soften and advocate some caution, like government control of AGI. 

Impacts on risk awareness, funding, and policy.

In futures like the above scenario, there will be widespread public calls for restrictions on developing and deploying AI. This should include pushes for restricting research on superintelligence. Restricting AI aimed at children and AI that can replace jobs will seem more urgent, and these might be useful for slowing AGI despite missing the main danger. One specific target might be AI that can learn to perform any job. That is general AI- the original definition of AGI. So this particular target of suspicion could help slow progress toward dangerous AGI and general superintelligence. 

It seems likely that decisions surrounding the alignment and use of proto-AGI systems will be made in a very different ambient environment. Crucial decisions will be made knowing that most people think ASI is very dangerous. This will push toward caution if diffusion of opinions has played a dominant role. But there's a chance that polarization and other types of motivated reasoning will cause those still working on AGI to insulate their beliefs about the difficulty of alignment. This seems likely to cut both ways. Maximizing the benefits seems tricky but worthwhile.

Increased funding for alignment is one main effect of increased alarm. More participation in anti-risk causes also seems inevitable. Many more people will probably experiment with alignment of LLM agents, for research, fun, and profit. 

Creating AGI in this world will still be a wildly irresponsible roll of the dice - but the bets will be made with different information and attitudes than we see now. Greater public concern should tilt the AI 2027 scenario at least somewhat toward success, and it could change the scenario sharply.

Concluding thoughts and questions

Many in AI safety have been dismayed that people are largely unmoved by abstract arguments for the dangers of AI. I think most people just haven't gotten interested enough yet. Those engaged so far usually work on or stand to benefit from AI, so they're biased to be dismissive of concerns. AI will be more immediately concerning as it takes jobs and seems more human - or alien. Conor Leahy frequently notes (e.g. [01:28:03] here), and I have observed, that nontechnical people usually understand the dangers of creating a thing more capable than you that might not be aligned with your interests. Their ancestors, after all, avoided being wiped out by misaligned humans.

So public opinion will shift, and it may shift dramatically and quickly. The extent to which this shift spreads to AI proponents, versus further motivating them to tune out the simple compelling logic and mounting evidence, remains to be predicted or seen.

Open questions:

• How much will opinions polarize between defenders and detractors of AI?
• How will deployment divide between generalist "entities" and task-specific agents?
• How much will AI companions and influencers fascinate younger generations?
• How much will fear amplify valid job-loss concerns?
• How effective will prosaic and System 2 Alignment techniques be at preventing obvious misalignments?
• How much will the question of real AGI alignment catch the public imagination?

The list could go on. Expansions and more factors are briefly discussed in this collapsible section.

Interacting factors in the growth of public concern about AI.

Technical capabilities and bottlenecks: Rate of progress on agent-specific challenges determines feasibility. There is progress underway now in late 2025 on all of the challenges that have plagued agents to date. Visual processing, poor reasoning, judgment, and situational awareness, access, authentication, and privacy, lack of memory/continuous learning (see my LLM AGI will have memory for review), speed, and compute costs have all been improved and can be expected to improve further - but what bottlenecks will remain is difficult to predict. These will have a major impact on how broadly LLM agents are deployed.

Tool vs. Actor AI: How soon and how strongly the "entity intuition" takes hold depends on the direction of progress and deployment. Specialized agents are currently more economically viable. But major labs are improving LLMs that are general reasoners, and specifically on their abilities for agentic roles. It seems likely we'll see mixes of both. If agents are largely specialized, do not learn, or reason creatively enough to make strange decisions, they will trigger fewer alignment concerns and existential fears. Agents competent enough to be useful without human supervision would be much more alarming, triggering entity intuitions even more strongly than "slave" agents. See the Rogue Replication addition to AI 2027 for much more.

Economic incentives and deployment patterns: Major labs focusing on coding/AI research agents may rush toward competent AGI. Startups are already scaffolding agents both for specialized roles and general learning/training (e.g. Tasklet), leveraging lab progress rather than competing with it. Routine tasks in customer service, data processing, and back-office operations provide lots of partial job replacement opportunities. Customer service deployments generate more widespread frustration with AI, but workers experience displacement and resentment regardless of visibility, and they will publicly complain.

Partial replacement dynamics: partial job replacement will make the risks of job loss less obvious and more arguable. It may also generate more human-AI friction than full automation. Workers spending their days supervising, correcting, and cleaning up after incompetent AI colleagues experience constant frustration while remaining under performance pressure. This irritation coulchd fan the flames of resentment over both real and feared job losses.

Regulatory and liability factors: Liability concerns (post-Air Canada precedent) may slow deployment in some jurisdictions while others race ahead. How much regulators respond with measured safety requirements vs. blanket restrictions will affect deployment. Regulatory fragmentation means deployment can proceed in permissive jurisdictions even if blocked elsewhere.

Cultural and demographic splits: Different groups will have radically different relationships with AI agents. Teenagers may embrace AI more strongly the more their parents' generation rejects it. This will create normalization in the next-generation workforce even as current workers resist. Professional and educational sector variation (tech-adjacent vs. craft professions) further fragments attitudes. Identity-linking to pro- or anti-AI stances accelerates polarization but also deepens engagement on both sides.

AI rights movements: I'll mention this as a factor, but I find it hard to guess how this will affect opinions and splits in opinion. Strong arguments might add fuel to entity intuitions, but weak ones could cause pushback and polarization. However, I expect rights and prohibition movements to be natural allies in the area of restricting general AI from taking jobs. And rights movements will anthropomorphize AI, which might be good, actually. 

Capability recognition vs. denialism: Whether people see steady improvement or persistent incompetence depends on usage frequency, professional incentives (threatened workers have motivated denialism), media diet, and which capabilities they track. Constant users see clear progress; one-time evaluators think "still just autocomplete." The public debate over capability trajectories is probably helpful—it forces attention to the obvious reality that capabilities will improve, even if the rate remains uncertain.

Psychology of alarm: Why incompetence generates alarm rather than dismissal remains an open question. Likely factors include: how entity-like the agents appear in their visible reasoning, whether failures look random or goal-directed, job threat salience, and media amplification patterns. Regardless of which psychological mechanisms dominate, plausible scenarios mostly increase public awareness and concern. The intensity and focus of that concern varies considerably with precise paths.

Improved alignment and control measures: Better safety measures reduce deployment friction and obvious misalignments, accelerating corporate adoption. Some system 2 alignment/internal control methods (see below) will also provide evidence of attempted misaligned actions in monitoring logs. A few insiders will blow whistles and publicize evidence of misalignment. Others will post anonymously, muddying the waters of real vs. fabricated alignment failures.

All of these factors interact. 

It's difficult to predict. But alignment plans should probably take into account the possibility of importantly different public opinion before we reach takeover-capable AGI.

Discuss

Published on November 7, 2025 4:40 PM GMT

Some podcasts are self-recommending on the ‘yep, I’m going to be breaking this one down’ level. This was very clearly one of those. So here we go.

As usual for podcast posts, the baseline bullet points describe key points made, and then the nested statements are my commentary.

If I am quoting directly I use quote marks, otherwise assume paraphrases.

The entire conversation takes place with an understanding that no one is to mention existential risk or the fact that the world will likely transform, without stating this explicitly. Both participants are happy to operate that way. I’m happy to engage in that conversation (while pointing out its absurdity in some places), but assume that every comment I make has an implicit ‘assuming normality’ qualification on it, even when I don’t say so explicitly.

On The Sam Altman Production Function

1. Cowen asks how Altman got so productive, able to make so many deals and ship so many products. Altman says people almost never allocate their time efficiently, and that when you have more demands on your time you figure out how to improve. Centrally he figures out what the core things to do are and delegates. He says deals are quicker now because everyone wants to work with OpenAI.
   1. Altman’s definitely right that most people are inefficient with their time.
   2. Inefficiency is relative. As in, I think of myself as inefficient with my time, and think of the ways I could be a lot more efficient.
   3. Not everyone responds to pressure by improving efficiency, far from it.
   4. Altman is good here to focus on delegation.
   5. It is indeed still remarkable how many things OpenAI is doing at once, with the associated worries about it potentially being too many things, and not taking the time to do them responsibly.

On Hiring Hardware People

1. What makes hiring in hardware different from in AI? Cycles are longer. Capital is more intense. So more time invested up front to pick wisely. Still want good, effective, fast-moving people and clear goals.
   1. AI seems to be getting pretty capital intensive?
2. Nvidia’s people ‘are less weird’ and don’t read Twitter. OpenAI’s hardware people feel more like their software people than they feel like Nvidia’s people.
   1. My guess is there isn’t a right answer but you need to pick a lane.
3. What makes Roon special? Lateral thinker, great at phrasing observations, lots of disparate skills in one place.
   1. I would add some more ingredients. There’s a sense of giving zero fucks, of having no filter, and having no agenda. Say things and let the chips fall.
   2. A lot of the disparate skills are disparate aesthetics, including many that are rare in AI, and taking all of them seriously at once.
4. Altman doesn’t tell researchers what to work on. Researchers choose, that’s it.
5. Email is very bad. Slack might not be good, it creates explosions of work including fake work to deal with, especially the first and last hours, but it is better than email. Altman suspects it’s time for a new AI-driven thing but doesn’t have it yet, probably due to lack of trying and unwillingness to pay focus and activation energy given everything else going on.
   1. I think email is good actually, and that Slack is quite bad.
   2. Email isn’t perfect but I like that you decide what you have ownership of, how you organize it, how you keep it, when you check it, and generally have control over the experience, and that you can choose how often you check it and aren’t being constantly pinged or expected to get into chat exchanges.
   3. Slack is an interruption engine without good information organization and I hate it so much, as in ‘it’s great I don’t have a job where I need slack.’
   4. There’s definitely room to build New Thing that integrates AI into some mix of information storage and retrieval, email slow communication, direct messaging and group chats, and which allows you to prioritize and get the right levels of interruption at the right times, and so on.
   5. However this will be tricky, you need to be ten times better and you can’t break the reliances people have. False negatives, where things get silently buried, can be quite bad.

On What GPT-6 Will Enable

1. What will make GPT-6 special? Altman suggests it might be able to ‘really do’ science. He doesn’t have much practical advice on what to do with that.
   1. This seems like we hit the wall of ‘…and nothing will change much’ forcing Altman to go into contortions.
   2. One thing we learned from GPT-5 is that the version numbers don’t have to line up with big capabilities leaps. The numbers are mostly arbitrary.

Tyler isn’t going to let him off that easy. At this point, I don’t normally do this, but exact words seem important so I’m going to quite the transcript.

COWEN: If I’m thinking about restructuring an entire organization to have GPT-6 or 7 or whatever at the center of it, what is it I should be doing organizationally, rather than just having all my top people use it as add-ons to their current stock of knowledge?

ALTMAN: I’ve thought about this more for the context of companies than scientists, just because I understand that better. I think it’s a very important question. Right now, I have met some orgs that are really saying, “Okay, we’re going to adopt AI and let AI do this.” I’m very interested in this, because shame on me if OpenAI is not the first big company run by an AI CEO, right?

COWEN: Just parts of it. Not the whole thing.

ALTMAN: No, the whole thing.

COWEN: That’s very ambitious. Just the finance department, whatever.

ALTMAN: Well, but eventually it should get to the whole thing, right? So we can use this and then try to work backwards from that. I find this a very interesting thought experiment of what would have to happen for an AI CEO to be able to do a much better job of running OpenAI than me, which clearly will happen someday. How can we accelerate that? What’s in the way of that? I have found that to be a super useful thought experiment for how we design our org over time and what the other pieces and roadblocks will be. I assume someone running a science lab should try to think the same way, and they’ll come to different conclusions.

COWEN: How far off do you think it is that just, say, one division of OpenAI is 85 percent run by AIs?

ALTMAN: Any single division?

COWEN: Not a tiny, insignificant division, mostly run by the AIs.

ALTMAN: Some small single-digit number of years, not very far. When do you think I can be like, “Okay, Mr. AI CEO, you take over”?

COWEN: CEO is tricky because the public role of a CEO, as you know, becomes more and more important.

1. On the above in terms of ‘oh no’:
   1. Oh no. Exactly the opposite. Shame on him if OpenAI goes first.
   2. OpenAI is the company, in this scenario, out of all the companies, we should be most worried about handing over to an AI CEO, for obvious reasons.
   3. If you’re wondering how the AIs could take over? You can stop wondering. They will take over because we will ask them to.
   4. CEO is an adversarial and anti-inductive position, where any weakness will be systematically exploited, and big mistakes can entirely sink you, and the way that you direct and set up the ‘AI CEO’ matters quite a lot in all this. The bar to a net positive AI CEO is much higher than the AI making on average better decisions, or having on average better features, and the actual bar is higher. Altman says ‘on the actual decision making maybe the AI is pretty good soon’ but this is a place where I’m going to be the Bottleneck Guy.
   5. CEO is also a position where, very obviously, misaligned means your company can be extremely cooked, and basically everything in it subverted, even if that CEO is a single human. Most of the ways in which this is limited are because the CEO can only be in one place at a time and do one thing at a time, couldn’t keep an eye on most things let alone micromanage them, and would require conspirators. A hostile AI CEO is death or subversion of the company.
   6. The ‘public role’ of the CEO being the bottleneck does not bring comfort here. If Altman (as he suggests) is public face and the AI ‘figures out what to do’ and Altman doesn’t actually get to overrule the AI (or is simply convinced not to) then the problem remains.
2. On the above in terms of ‘oh yeah’:
   1. There is the clear expectation from both of them that AI will rise, reasonably soon, to the level of at least ‘run the finance department of a trillion dollar corporation.’ This doesn’t have to be AGI but it probably will be, no?
   2. It’s hard for me to square ‘AIs are running the actual decision making at top corporations’ with predictions for only modest GDP growth. As Altman notes, the AI CEO needs to be a lot better than the human CEO in order to get the job.
   3. They are predicting billion-dollar 2-3 person companies, with AIs, within three years.
3. Altman asks potential hires about their use of AI now to predict their level of AI adoption in the future, which seems smart. Using it as ‘better Google’ is a yellow flag, thinking about day-to-day in three years is a green flag.
4. In three years Altman is aiming to have a ‘fully automated AI researcher.’ So it’s pretty hard to predict day-to-day use in three years.

On government backstops for AI companies

A timely section title.

1. Cowen and Altman are big fans of nuclear power (as am I), but people worry about them. Cowen asks, do you worry similarly about AI and the similar Nervous Nellies, even if ‘AI is pretty safe’? Are the Feds your insurer? How will you insure everything?
   1. Before we get to Altman’s answer can we stop to think about how absolutely insane this question is as presented?
   2. Cowen is outright equating worries about AI to worries about nuclear power, calling both Nervous Nellies. My lord.
   3. The worry about AI risks is that the AI companies might be held too accountable? Might be asked to somehow provide too much insurance, when there is clearly no sign of any such requirement for the most important risks? They are building machines that will create substantial catastrophic and even existential risks, massive potential externalities.
   4. And you want the Federal Government to actively insure against AI catastrophic risks? To say that it’s okay, we’ve got you covered? This does not, in any way, actually reduce the public’s or world’s exposure to anything, and it further warps company incentives. It’s nuts.
   5. Not that even the Federal Government can actually ensure us here even at our own expense, since existential risk or sufficiently large catastrophic or systemic risk also wipes out the Federal Government. That’s kind of the point.
   6. The idea that the people are the Nervous Nellies around nuclear, which has majority public support, while Federal Government is the one calming them down and ensuring things can work is rather rich.
   7. Nuclear power regulations are insanely restrictive and prohibitive, and the insurance the government writes does not substantially make up for this, nor is it that expensive or risky. The NRC and other regulations are the reason we can’t have this nice thing, in ways that don’t relate much if at all to the continued existence of these Nervous Nellies. Providing safe harbor in exchange of that really is the actual least you can do.
   8. AI regulations impose very few rules and especially very few safety rules.
   9. Yes, there is the counterpoint that AI has to follow existing rules and thus is effectively rather regulated, but I find this rather silly as an argument, and no I don’t think the new laws around AI in particular move that needle much.
2. Altman points out the Federal Government is the insurer of last resort for anything sufficiently large, whether you want it to be or not, but no not in the way of explicitly writing insurance policies.
   1. I mean yes if AI crashes the economy or does trillions in damages or what not, then the Federal Government will have to try and step in. This is a huge actual subsidy to the AI companies and they should (in theory anyway) be pay for it.
   2. A bailout for the actual AI companies if they are simply going bankrupt? David Sacks has made it clear our answer is no thank you, and rightfully so. Obviously, at some point the Fed Put or Trump Put comes into play in the stock market, that ship has sailed, but no we will not save your loans.
   3. And yeah, my lord, the idea that the Feds would write an insurance policy.
3. Cowen then says he is worried about the Feds being the insurer of first resort and he doesn’t want that, Altman confirms he doesn’t either and doesn’t expect it.
   1. It’s good that they don’t want this to happen but this only slightly mitigates my outrage at the first question and the way it was presented.
4. Cowen points out Trump is taking equity in Intel, lithium and rare earths, and asks how this applies to OpenAI. Altman mostly dodges, pivots to potential loss of meaning in the world, and points out the government might have strong opinions about AI company actions.
   1. Cowen doesn’t say it here but to his credit is on record correctly opposing this taking of equity in companies correctly identifying it as ‘seizing the means of production’ and pointing out it is the wrong tool for the job.
   2. This really was fully a non-answer. I see why that might be wise.
   3. Could OpenAI be coerced into giving up equity, or choose to do so as part of a regulatory capture play? Yeah. It would be a no-good, very bad thing.
   4. The government absolutely will and needs to have strong opinions about AI company actions and set the regulations and rules in place and otherwise play the role of being the actual government.
   5. If the government does not govern the AI companies, then the government will wake up one day to find the AI companies have become the government.

On monetizing AI services

1. Tyler Cowen did a trip through France and Spain and booked all but one hotel with GPT-5 (not directly in the app), and almost every meal they ate, and Altman didn’t get paid for that. Shouldn’t he get paid?
   1. Before I get to Altman’s answer, I will say that for specifically Tyler this seems very strange to me, unless he’s running an experiment as research.
   2. As in, Tyler has very particular preferences and a lot of comparative advantage in choosing hotels and especially restaurants, especially for himself. It seems unlikely that he can’t do better than ChatGPT?
   3. I expect to be able to do far better than ChatGPT on finding restaurants, although with a long and highly customized prompt, maybe? But it would require quite a lot of work.
   4. For hotels, yeah, I think it’s reasonably formulaic and AI can do fine.
2. Altman responds that often ChatGPT is cited as the most trusted tech product from a big tech company. He notes that this is weird given the hallucinations. But it makes sense in that it doesn’t have ads and is in many visible ways more fully aligned with user preferences than other big tech products that involve financial incentives. He notes that a transaction fee probably is fine but any kind of payment for placement would endanger this.
   1. ChatGPT being most trusted is definitely weird given it is not very reliable.
   2. It being most trusted is an important clue to how people will deal with AI systems going forward, and it should worry you in important ways.
   3. In particular, trust for many people is about ‘are they Out To Get You?’ rather than reliability or overall quality, or are expectations set fairly. Compare to the many people who otherwise trust a Well Known Liar.
   4. I strongly agree with Altman about the payola worry, as Cowen calls it. Cowen says he’s not worried about it, but doesn’t explain why not.
   5. OpenAI’s instant checkout offerings and policies are right on the edge on this. I think in their present form they will be fine but they’re on thin ice.
3. Cowen’s worry is that OpenAI will have a cap on how much commission they can charge, because stupider services will then book cheaply if you charge too much. Altman says he expects much lower margins.
   1. AI will as Altman notes make many markets much more efficient by vastly lowering search costs and transaction costs, which will lower margins, and this should include commissions.
   2. I still think OpenAI will be able to charge substantial commissions if it retains its central AI position with consumers, for the same reason that other marketplaces have not lost their ability to extract commissions, including some very large ones. Every additional hoop you ask a customer to go through loses a substantial portion of sales. OpenAI can pull the same tricks as Steam and Amazon and Apple including on price parity, and many will pay.
   3. This is true even if there are stupider services that can do the booking and are generally 90% as good, so long as OpenAI is the consumer default.
4. Cowen doubles down on this worry about cheap competing agents, Altman notes that hotel booking is not the way to monetize, Cowen says but of course you do want to do that, Altman says no he wants to do new science, but ChatGPT and hotel booking is good for the world.
   1. This feels like a mix of a true statement and a dishonest dodge.
   2. As in, of course he wants to do hotel booking and make money off it, it’s silly to pretend that you don’t and there’s nothing wrong with that. It’s not the main goal, but it drives growth and valuation and revenue all of which is vital to the AGI or science mission (whether you agree with that mission or not).
5. Cowen asks, you have a deal coming with Walmart, if you were Amazon would you make a deal with OpenAI or fight back? Altman says he doesn’t know, but that if he was Amazon he would fight back.
   1. Great answer from Altman.
   2. One thing Altman does well is being candid in places you would not expect, where it is locally superficially against his interests, but where it doesn’t actually cost him much. This is one of those places.
   3. Amazon absolutely cannot fold here because it loses too much control over the customer and customer flow. They must fight back. Presumably they should fight back together with their friends at Anthropic?
6. Cowen asks about ads. Altman says some ads would be bad as per earlier, but other kinds of ads would be good although he doesn’t know what the UI is.
   1. Careful, Icarus.
   2. There definitely are ‘good’ ways to do ads if you keep them entirely distinct from the product, but the temptations and incentives here are terrible.

On AI’s future understanding of intangibles

1. What should OpenAI management know about KSA and UAE? Altman says it’s mainly knowing who will run the data centers and what security guarantees they will have, with data centers being built akin to US embassies or military bases. They bring in experts and as needed will bring in more.
   1. I read this as a combination of outsourcing the worries and not worrying.
   2. I would be more worried.
2. Cowen asks, how good will GPT-6 be at teaching these kinds of national distinctions, or do you still need human experts? Altman expects to still need the experts, confirms they have an internal eval for that sort of thing but doesn’t want to pre-announce.
   1. My anticipation is that GPT-6 and its counterparts will actually be excellent at understanding these country distinctions in general, when it wants to be.
   2. My anticipation is also that GPT-6 will be excellent at explaining things it knows to humans and helping those humans learn, when it wants to, and this is already sufficiently true for current systems.
   3. The question is, will you be able to translate that into learning and understanding such issues?
   4. Why is this uncertain? Two concerns.
   5. The first concern is that understanding may depend on analysis of particular key people and relationships, in ways that are unavailable to AI, the same way you can’t get them out of reading books.
   6. The second concern is that to actually understand KSA and UAE, or any country or culture in general, requires communicating things that it would be impolitic to say out loud, or for an AI to typically output. How do you pass on that information in this context? It’s a problem.
3. Cowen asks about poetry, predicts you’ll be able to get the median Pablo Neruda poem but not the best, maybe you’ll get to 8.8/10 in a few years. Altman says they’ll reach 10/10 and Cowen won’t care, Cowen promises he’ll care but Altman equates it to AI chess players. Cowen responds there’s something about a great poem ‘outside the rubric’ and he worries humans that can’t produce 10s can’t identify 10s? Or that only humanity collectively and historically can decide what is a 10?
   1. This is one of those ‘AI will never be able to [X] at level [Y]’ claims so I’m on Altman’s side here, a sufficiently capable AI can do 10/10 on poems, heck it can do 11/10 on poems. But yeah, I don’t think you or I will care other than as a technical achievement.
   2. If an AI cannot produce sufficiently advanced poetry, that means that the AI is insufficiently advanced. Also we should not assume that future AIs or LLMs will share current techniques or restrictions. I expect innovation with respect to poetry creation.
   3. The thing being outside the rubric is a statement primarily about the rubric.
   4. If only people writing 10s can identify 10s then for almost all practical purposes there’s no difference between a 9 and a 10. Why do we care, if we literally can’t tell the difference? Whereas if we can tell the difference, if verification is easier than generation as it seems like it should be here, then we can teach the AI how to tell the difference.
   5. I think Cowen is saying that a 10-poem is a 9-poem that came along at the right time and got the right cultural resonance, in which case sure, you cannot reliably produce 10s, but that’s because it’s theoretically impossible to do that, and no human could do that either. Pablo Neruda couldn’t do it.
   6. As someone who has never read a poem by Pablo Neruda, I wanted to see what this 10.0 business was all about, so by Claude’s recommendation of ‘widely considered best Neruda poem’ without any other context, I selected Tonight I Can Write (The Saddest Lines). And not only did it not work on me, it seemed like something an AI totally could write today, on the level of ‘if you claimed to have written this in 2025 I’d have suspected an AI did write it.’
   7. With that in mind, I gave Claude context and it selected Ode to the Onion. Which also didn’t do anything for me, and didn’t seem like anything that would be hard for an AI to write. Claude suggests it’s largely about context, that this style was new at the time, and I was reading translations into English and I’m no poetry guy, and agrees that in 2025 yes an AI could produce a similar poem, it just wouldn’t land because it’s no longer original.
   8. I’m willing to say that whatever it is Tyler thinks AI can’t do, also is something I don’t have the ability to notice. And which doesn’t especially motivate me to care? Or maybe is what Tyler actually wants something like ‘invent new genre of poetry’?
   9. We’re not actually trying to get AIs to invent new genres of poetry, we’re not trying to generate the things that drive that sort of thing, so who is to say if we could do it. I bet we could actually. I bet somewhere in the backrooms is a 10/10 Claude poem, if you have eyes to see.

On Chip-Building

1. It’s hard. Might get easier with time, chips designing chips.
2. Why not make more GPUs? Altman says, because we need more electrons. What he needs most are electrons. We’re working hard on that. For now, natural gas, later fusion and solar. He’s still bullish on fusion.
   1. This ‘electrons’ thing is going to drive me nuts on a technical level. No.
   2. This seems simply wrong? We don’t build more GPUs because TSMC and other bottlenecks mean we can’t produce more GPUs.
   3. That’s not to say energy isn’t an issue but the GPUs sell out.
   4. Certainly plenty of places have energy but no GPUs to run with them.
3. Cowen worries that fusion uses the word ‘nuclear.’
   1. I don’t. I think that this is rather silly.
   2. The problem with fusion is purely that it doesn’t work. Not yet, anyway.
   3. Again, the people are pro-nuclear power. Yay the people.
4. Cowen asks do you worry about a scenario where superintelligence does not need much compute, so you’re betting against progress over a 30-year time horizon?
   1. Always pause when you hear such questions to consider that perhaps under such a scenario this is not the correct thing to worry about?
   2. As in, if we not only have superintelligence it also does not need so much compute, the last thing I am going to ponder next is the return on particular investments of OpenAI, even if I am the CEO of OpenAI.
   3. If we have sufficiently cheap superintelligence that we have both superintelligence and an abundance of compute, ask not how the stock does, ask questions like how the humans survive or stay in control at all, notice that the entire world has been transformed, don’t worry about your damn returns.
5. Altman responds if compute is cheaper people will want more. He’ll take that bet every day, and the energy will still be useful no matter the scenario.
   1. Good bet, so long as it matters what people want.
6. Cowen loves Pulse, Altman says people love Pulse, the reason you don’t hear more is it’s only available to Pro users. Altman uses Pulse for a combination of work related news and family opportunities like hiking trails.
   1. I dabble with Pulse. It’s… okay? Most of the time it gives me stories I already know about, but occasionally there’s something I otherwise missed.
   2. I’ve tried to figure out things it will be good at monitoring, but it’s tough, maybe I should invest more time in giving it custom instructions.
   3. In theory it’s a good idea.
   4. It suffers from division of context, since the majority of my recent LLM activity has been on Claude and perhaps soon will include Gemini.

On Sam’s outlook on health, alien life, and conspiracy theories

Ooh, fun stuff.

1. What is Altman’s nuttiest view about his own health? Altman says he used to be more disciplined when he was less busy, but now he eats junk food and doesn’t exercise enough and it’s bad. Whereas before he once got in the hospital for trying semaglutide before it was cool, which itself is very cool.
   1. There’s weird incentives here. When you have more going on it means you have less time to care about food and exercise but also makes it more important.
   2. I’d say that over short periods (like days and maybe weeks) you can and should sacrifice health focus to get more attention and time on other things.
   3. However, if you’re going for months or years, you want to double down on health focus up to some reasonable point, and Altman is definitely here.
   4. That doesn’t mean obsess or fully optimize of course. 80/20 or 90/10 is good.
2. Cowen says junk food doesn’t taste good and good sushi tastes better, Altman says yes junk food tastes good and sometimes he wants a chocolate chip cookie at 11:30 at night.
   1. They’re both right. Sometimes you want the (fresh, warm, gooey) chocolate chip cookie and not the sushi, sometimes you want the sushi and not the cookie.
   2. You get into habits and your body gets expectations, and you develop a palate.
   3. With in-context unlimited funds you do want to be ‘spending your calories’ mostly on the high Quality things that are not junk, but yeah in the short term sometimes you really want that cookie.
   4. I think I would endorse that I should eat 25% less carbs and especially ‘junk’ than I actually do, maybe 50%, but not 75% less, that would be sad.
3. Cowen asks if there’s alien life on the moons of Saturn, says he does believe this. Altman says he has no opinion, he doesn’t know.
   1. I’m actually with Altman in the sense that I’m happy to defer to consensus on the probability here, and I think it’s right not to invest in getting an opinion, but I’m curious why Cowen disagrees. I do think we can be confident there isn’t alien life there that matters to us.
4. What about UAPs? Altman thinks ‘something’s going on there’ but doesn’t know, and doubts it’s little green men.
   1. I am highly confident it is not little green men. There may or may not be ‘something going on’ from Earth that is driving this, and my default is no.
5. How many conspiracy theories does Altman believe in? Cowen says zero, at least in the United States. Altman says he’s predisposed to believe, has an X-Files ‘I want to believe’ t-shirt, but still believes in either zero or very few. Cowen says he’s the opposite, he doesn’t want to believe, maybe the White Sox fixed the World Series way back when, Altman points out this doesn’t count.
   1. The White Sox absolutely fixed that 1919 World Series, we know this. At the time it was a conspiracy theory but I think that means this is no longer a conspiracy theory?
   2. I also believe various other sporting events have been fixed, but with less certainty, and to varying degrees – sometimes there’s an official’s finger on the scale but the game is real, other times you’re in Russia and the players literally part the seas to ensure the final goal is scored, and everything in between, but most games played in the West are on or mostly on the level.
   3. Very obviously there exist conspiracies, some of which succeed at things, on various scales. That is distinct from ‘conspiracy theory.’
   4. As a check, I asked Claude for the top 25 most believed conspiracy theories in America. I am confident that 24 out of the 25 are false. The 25th was Covid-19 lab origins, which is called a conspiracy theory but isn’t one. If you modify that to ‘Covid-19 was not only from a lab but was released deliberately’ then I’m definitely at all 25 are false.
6. Cowen asks again, how would you revitalize St. Louis with a billion dollars and copious free time? Altman says start a Y-Combinator thing, which is pretty similar to what Altman said last time. But he suggests that’s because that would be Altman’s comparative advantage, someone else would do something else.
   1. This seems correct to me.

On regulating AI agents

1. Should it be legal to release an AI agent into the wild, unowned, untraceable? Altman says it’s about thresholds. Anything capable of self-replication needs oversight, and the question is what is your threshold.
   1. Very obviously it should not be legal to, without checking first, release a self-replicating untraceable unowned highly capable agent into the wild that we have no practical means of shutting down.
   2. As a basic intuition pump, you should be responsible for what an AI agent you release into the wild does the same way you would be if you were still ‘in control’ of that agent, or you hired the agent, or if you did the actions yourself. You shouldn’t be able to say ‘oh that’s not on me anymore.’
   3. Thus, if you cannot be held accountable for it, I say you can’t release it. A computer cannot be held accountable, therefore a computer cannot make a management decision, therefore you cannot release an agent that will then make unaccountable management decisions.
   4. That includes if you don’t have the resources to take responsibility for the consequences, if they rise to the level where taking all your stuff and throwing you in jail is not good enough. Or if the effects cannot be traced.
   5. Certainly if such an agent poses a meaningful risk of loss of human control or of catastrophic or existential risks, the answer needs to be a hard no.
   6. If what you are doing is incompatible with such agents not being released into the wild, then what you are doing, via backchaining, is also not okay.
   7. There presumably should be a method whereby you can do this legally, with some set of precautions attached to it.
   8. Under what circumstances an open weight model would count as any of this is left as an open ended question.
2. What to do if it happens and you can’t turn it off? Ring-fence it, identify, surveil, sanction the host location? Altman doesn’t know, it’s the same as the current version of this problem, more dangerous but we’ll have better defenses, and we need to urgently work on this problem.
   1. I don’t disagree with that response but it does not indicate a good world state.
   2. It also suggests the cost of allowing such releases is currently high.

On new ways to interface with AI

1. Both note (I concur) that it’s great to read your own AI responses but other people’s responses are boring.
   1. I do sometimes share AI queries as a kind of evidence, or in case someone needs a particular thing explained and I want to lower activation energy on asking the question. It’s the memo you hope no one ever needs to read.
2. Altman says people like watching other people’s AI videos.
   1. Do they, though?
3. Altman points out that everyone having great personal AI agents is way more interesting than all that, with new social dynamics.
   1. Indeed.
   2. The new social dynamics include ‘AI runs the social dynamics’ potentially along with everything else in short order.
4. Altman’s goal is a new kind of computer with an AI-first interface very different from the last 50 years of computing. He wants to question basic assumptions like an operating system or opening a window, and he does notice the skulls along the ‘design a new type of computer’ road. Cowen notes that people really like typing into boxes.
   1. Should AI get integrated into computers far more? Well, yeah, of course.
   2. How much should this redesign the computer? I’m more skeptical here. I think we want to retain control, fixed commands that do fixed things, the ability to understand what is happening.
   3. In gaming, Sid Meier called this ‘letting the player have the fun.’ If you don’t have control or don’t understand what is happening and how mechanics work, then the computer has all the fun. That’s no good, the player wants the fun.
   4. Thus my focus would be, how do we have the AI enable the user to have the fun, as in understand what is happening and direct it and control it more when they want to? And also to enable the AI to automate the parts the user doesn’t want to bother about?
   5. I’d also worry a lot about predictability and consistently across users. You simultaneously want the AI to customize things to your preferences, but also to be able to let others share with you the one weird trick or explain how to do a thing.

On how normies will learn to use AI

1. What would an ideal partnership with a university look like? Altman isn’t sure, maybe try 20 different experiments. Cowen worries that higher education institutions lack internal reputational strength or credibility to make any major changes and all that happens is privatized AI use, and Altman says he’s ok with it.
   1. It does seem like academia and universities in America are not live players, they lack the ability to respond to AI or other changes, and they are mostly going to collect what rents they can until they get run over.
   2. In some senses I agree This Is Fine, obviously it is a huge tragedy all the time and money being wasted but there is not much we can do about this and it will be increasingly viable to bypass the system, or to learn in spite of it.
2. How will the value of a typical college degree change in 5-10 years? Cowen notes it’s gone down in the last 10, after previously going up. Altman says further decline, faster than before, but not to zero as fast as it should.
   1. Sounds right to me under an ‘economic normal’ scenario.
3. So what does get returns other than learning AI? Altman says yes, wide benefits to learning to use AI well, including but not limited to things like new science or starting companies.
   1. I notice Altman didn’t name anything non-AI that goes up in value.
   2. I don’t think that’s because he missed a good answer. Ut oh.
4. How do you teach normies to use AI five years from now, for their own job? Altman says basically people learn on their own.
   1. It’s great that they can learn on their own, but this definitely is not optimal.
   2. As in, you should be able to do a lot better by teaching people?
   3. There’s definitely a common theme of lack of curiosity, where people need pushes in the right directions. Perhaps AI itself can help more with this.
5. Will we still read books? Altman notes books have survived a lot of things.
   1. Books are on rapid decline already though. Kids these days, AIUI, read lots of text, but basically don’t read books.
6. Will we start creating our own movies? What else will change? Altman says how we use emails and calls and meetings and write documents will change a lot, family time or time in nature will change very little.
   1. There’s the ‘economic normal’ and non-transformational assumption here, that the outside world looks the same and it’s about how you personally interact with AIs. Altman and Cowen both sneak this in throughout.
   2. Time with family has changed a lot in the last 50-100 years. Phones, computers and television, even radio, the shift in need for various household activities, cultural changes, things like that. I expect more change here, even if in some sense it doesn’t change much, and even if those who are wisest in many ways let it change the least, again in these ‘normal’ worlds.
   3. All the document shuffling, yes, that will change a lot.
   4. Altman doesn’t take the bait on movies and I think he’s mostly right. I mostly don’t want customized movies, I want to draw from the same movies as everyone else, I want to consume someone’s particular vision, I want a fixed document.
   5. Then again, we’ve moved into a lot more consumption of ephemeral, customized media, especially short form video, mostly I think this is terrible, and (I believe Cowen agrees here) I think we should watch more movies instead, I would include television.
   6. I think there’s a divide. Interactive things like games and in the future VR, including games involving robots or LLM characters, are a different kind of experience that should often be heavily customizable. There’s room for personalized, unique story generation, and interactions, too.

On AI’s effect on the price of housing and healthcare

1. Will San Francisco, at least within the West, remain the AI center? Altman says this is the default, and he loves the Bay Area and thinks it is making a comeback.
2. What about housing costs? Can AI make them cheaper? Altman thinks AI can’t help much with this.
   1. Other things might help. California’s going at least somewhat YIMBY.
   2. I do think AI can help with housing quite a lot, actually. AI can find the solutions to problems, including regulations, and it can greatly reduce ‘transaction costs’ in general and reduce the edge of local NIMBY forces, and otherwise make building cheaper and more tractable.
   3. AI can also potentially help a lot with political dysfunction, institutional design, and other related problems, as well as to improve public opinion.
   4. AI and robotics could greatly impact space needs.
   5. Or, of course, AI could transform the world more generally, including potentially killing everyone. Many things impact housing costs.
3. What about food prices? Altman predicts down, at least within a decade.
   1. Medium term I’d predict down for sure at fixed quality. We can see labor shift back into agriculture and food, probably we get more highly mechanized agriculture, and also AI should optimize production in various ways.
   2. I’d also predict people who are wealthier due to AI invest more in food.
   3. I wouldn’t worry about energy here.
4. What about healthcare? Cowen predicts we will spend more and live to 98, and the world will feel more expensive because rent won’t be cheaper. Altman disagrees, says we will spend less on healthcare, we should find cures and cheap treatments, including through pharmaceuticals and devices and also cheaper delivery of services, whereas what will go up in price are status goods.
   1. There’s two different sets of dynamics in healthcare I think?
   2. In the short run, transaction costs go down, people get better at fighting insurance companies, better at identifying and fighting for needed care. Demand probably goes up, total overall real spending goes up.
   3. Ideally we would also be eliminating unnecessary, useless or harmful treatments along the way, and thus spending would go down, since much of our medicine is useless, but alas I mostly don’t expect this.
   4. We also should see large real efficiency gains in provision, which helps.
   5. Longer term (again, in ‘normal’ worlds), we get new treatments, new drugs and devices, new delivery systems, new understanding, general improvement, including making many things cheaper.
   6. At that point, lots of questions come into play. We are wealthier with more to buy, so we spend more. We are wiser and know what doesn’t work and find less expensive solutions and gain efficiency, so we spend less. We are healthier so we spend less now but live longer which means we spend more.
   7. In the default AGI scenarios, we don’t only live to 98, we likely hit escape velocity and live indefinitely, and then it comes down to what that costs.
   8. My default in the ‘good AGI’ scenarios is that we spend more on healthcare in absolute terms, but less as a percentage of economic capacity.

On reexamining freedom of speech

1. Cowen asks if we should reexamine patents and copyright? Altman has no idea.
   1. Our current systems are obviously not first best, already were not close.
   2. Copyright needs radical rethinking, and already did. Terms are way too long. The ‘AI outputs have no protections’ rule isn’t going to work. Full free fair use for AI training is no good, we need to compensate creators somehow.
   3. Patents are tougher but definitely need rethinking.
2. Cowen is big on freedom of speech and worries people might want to rethink the First Amendment in light of AI.
   1. I don’t see signs of this? I do see signs of people abandoning support for free speech for unrelated reasons, which I agree is terrible. Free speech will ever and always be under attack.
   2. What I mostly have seen are attempts to argue that ‘free speech’ means various things in an AI context that are clearly not speech, and I think these should not hold and that if they did then I would worry about taking all of free speech down with you.
3. They discuss the intention to expand free expression of ChatGPT, the famous ‘erotica tweet.’ Perhaps people don’t believe in freedom of expression after all? Cowen does have that take.
   1. People have never been comfortable with actual free speech, I think. Thus we get people saying things like ‘free speech is good but not [misinformation / hate speech / violence or gore / erotica / letting minors see it / etc].’
   2. I affirm that yes LLMs should mostly allow adults full freedom of expression.
   3. I do get the issue in which if you allow erotica then you’re doing erotica now, and ChatGPT would instantly become the center of erotica and porn, especially if the permissions expand to image and even video generation.
4. Altman wants to change subpoena power with respect to AI, to allow your AI to have the same protections as a doctor or lawyer. He says America today is willing to trust AI on that level.
   1. It’s unclear here if Altman wants to be able to carve out protected conversations for when the AI is being a doctor or lawyer or similar, or if he wants this for all AI conversations. I think it is the latter one.
   2. You could in theory do the former, including without invoking it explicitly, by having a classifier ask (upon getting a subpoena) whether any given exchange should qualify as privileged.
   3. Another option is to ‘hire the AI lawyer’ or other specialist by paying a nominal fee, the way lawyers will sometimes say ‘pay me a dollar’ in order to nominally be your lawyer and thus create legal privilege.
   4. There could also be specialized models to act as these experts.
   5. But also careful what you wish for. Chances seem high that getting these protections would come with obligations AI companies do not want.
   6. The current rules for this are super weird in many places, and the result of various compromises of different interests and incentives and lobbies.
   7. What I do think would be good at a minimum is if ‘your AI touched this information’ did not invalidate confidentiality, whereas third party sharing of information often will do invalidate confidentiality.
   8. Google search is a good comparison point because it ‘feels private’ but your search for ‘how to bury a body’ very much will end up in your court proceeding. I can see a strong argument that your AI conversations should be protected but if so then why not your Google searches?
   9. Similarly, when facing a lawsuit, if you say your ChatGPT conversations are private, do you also think your emails should be private?

On humanity’s persuadability

1. Cowen asks about LLM psychosis. Altman says it’s a ‘very tiny thing’ but not a zero thing, which is why the restrictions put in place in response to it pissed users off, most people are okay so they just get annoyed.
   1. Users always get annoyed by restrictions and supervision, and the ones that are annoyed are often very loud.
   2. The actual outright LLM psychosis is rare but the number of people who actively want sycophancy and fawning and unhealthy interactions, and are mostly mad about not getting enough of that, are very common.

I’m going to go full transcript here again, because it seems important to track the thinking:

ALTMAN: Someone said to me once, “Never ever let yourself believe that propaganda doesn’t work on you. They just haven’t found the right thing for you yet.” Again, I have no doubt that we can’t address the clear cases of people near a psychotic break.

For all of the talk about AI safety, I would divide most AI thinkers into these two camps of “Okay, it’s the bad guy uses AI to cause a lot of harm,” or it’s, “the AI itself is misaligned, wakes up, whatever, intentionally takes over the world.”

There’s this other category, third category, that gets very little talk, that I think is much scarier and more interesting, which is the AI models accidentally take over the world. It’s not that they’re going to induce psychosis in you, but if you have the whole world talking to this one model, it’s not with any intentionality, but just as it learns from the world in this continually coevolving process, it just subtly convinces you of something. No intention, it just does. It learned that somehow. That’s not as theatrical as chatbot psychosis, obviously, but I do think about that a lot.

COWEN: Maybe I’m not good enough, but as a professor, I find people pretty hard to persuade, actually. I worry about this less than many of my AI-related friends do.

ALTMAN: I hope you’re right.

1. On Altman’s statement:
   1. The initial quote is wise.
   2. The division into these three categories is a vast oversimplification, as all such things are. That doesn’t make the distinction not useful, but I worry about it being used in a way that ends up being dismissive.
   3. In particular, there is a common narrowing of ‘the AI itself is misaligned’ into ‘one day it wakes up and takes over the world’ and then people think ‘oh okay all we have to do is ensure that if one day one of them wakes up it doesn’t get to take over the world’ or something like that. The threat model within the category is a lot broader than that.
   4. There’s also ‘a bunch of different mostly-not-bad guys use the AI to pursue their particular interests, and the interactions and competitions and evolutions between them go badly or lead to loss of human control’ and there’s ‘we choose to put the AIs in charge of the world on purpose’ with or without AI having a hand in that decision, and so on and so forth.
   5. On the particular worry here of Altman’s, yes, I think that extended AI conversations are very good at convincing people of things, often in ways no one (including the AI) intended, and as AIs gain more context and adjust to it more, as they will, this will become a bigger and more common thing.
   6. People are heavily influenced by, and are products of, their environment, and of the minds they interact with on a regular basis.
2. On Cowen’s statement:
   1. A professor is not especially well positioned to be persuasive, nor does a professor typically get that much time with engaged students one-on-one.
   2. When people talk about people being ‘not persuadable’ they typically talk about cases where people’s defenses are relatively high, in limited not-so-customized interactions in which the person is not especially engaged or following their curiosity or trusting, and where the interaction is divorced from their typical social context.
   3. We have very reliable persuasion techniques, in the sense that for the vast majority of human history most people in each area of the world believed in the local religion and local customs and were patriots of the local area and root for the local sports team and support the local political perspectives, and so on, and were persuaded to pass all that along to their own children.
   4. We have a reliable history of armies being able to break down and incorporate new people, of cults being able to do so for new recruits, for various politicians to often be very convincing and the best ones to win over large percentages of people they interact with in person, for famous religious figures to be able to do massive conversions, and so on.
   5. Marxists were able to persuade large percentages of the world, somehow.
   6. Children who attend school and especially go to college tend to exit with the views of those they attend with, even when it conflicts with their upbringing.
   7. If you are talking to an AI all the time, and it has access to your details and stuff, this is very much an integrated social context, so yes many are going over time to be highly persuadable.
   8. This is all assuming AI has to stick to Ordinary Human levels of persuasiveness, which it won’t have to.
   9. There are also other known techniques to persuade humans that we will not be getting into here, that need to be considered in such contexts.
   10. Remember the AI box experiments.
   11. I agree that if we’re talking about ‘the AI won’t in five minutes be able to convince you to hand over your bank account information’ that this will require capabilities we don’t know about, but that’s not the threshold.
3. If you have a superintelligence ready to go, that is ‘safety-tested,’ that’s about to self-improve, and you get a prompt to type in, what do you type? Altman raises this question, says he doesn’t have an answer but he’s going to have someone ask the Dalai Lama.
   1. I also do not know the right answer.
   2. You’d better know that answer well in advance.

Discuss

Published on November 7, 2025 4:25 PM GMT

In a preprint from October 13, two researchers from the Ruhr University Bochum and the University of Bonn in Germany found that while leading AI companies say they will design their most general-purpose AI, often called AGI, based on the most stringent safety principles—adapted from fields like nuclear engineering—the safety techniques they apply do not satisfy them.

In particular, the authors note that existing proposals fail to satisfy the principle known as defense in depth, which calls for the application of multiple, redundant, and independent safety mechanisms. The conventional safety methods that companies are known to apply are not independent; in certain problematic scenarios, which are relatively easy to foresee, they all tend to fail simultaneously.

Many leading AI companies, including Anthropic, Microsoft, and OpenAI have all published safety documents that explicitly mention their intention to implement defense in depth for the design of their most advanced AI systems. 

In an interview with Foom, the first co-author of the study, Leonard Dung of the Ruhr University Bochum, said that it was not surprising that many of the methods for designing AI systems to be safe might fail. Research on making powerful AI systems safe is broadly viewed to be at an early stage of maturity.

More surprising for Dung, and also concerning, was that it was him and his co-author, who are academic scholars in philosophy and machine learning, to make what is arguably a foundational contribution to the safety literature of a new branch of industrial engineering.

"There has not been much systematic thinking about what exactly does it mean to take a defense-in-depth approach to safety," said Dung. "The sort of basic way of thinking about risk that you would expect these companies—and policymakers who regulate these companies—to implement has not been implemented." 

Continue reading at foommagazine.org ...

Discuss

Published on November 7, 2025 4:19 PM GMT

Over the past year, I have talked to several people about whether they expect frontier AI companies to transition away from the current paradigm of transformer LLMs toward models that reason in neuralese within the next few years. This post summarizes 13 common arguments I’ve heard, six in favor and seven against a transition to neuralese AIs. The following table provides a summary:

Arguments for a transition to neuraleseArguments against a transition to neuraleseA lot of information gets lost in text bottlenecks.Natural language reasoning might be a strong local optimum that takes a lot of training effort to escape.The relative importance of post-training compared to pre-training is increasing.Recurrent LLMs suffer from a parallelism trade-off that makes their training less efficient.There’s an active subfield researching recurrent LLMs.There’s significant business value in being able to read a model’s CoTs.Human analogy: natural language might not play that big of a role in human thinking.Human analogy: even if natural language isn’t humans’ primary medium of thought, we still rely on it a lot.SGD inductive biases might favor directly learning good sequential reasoning algorithms in the weight space.Though significant effort has been spent on getting neuralese models to work, we still have none that work at scale.In the limit of capabilities, neuralese is strictly more powerful than CoT reasoning.There’s a depth-latency trade-off that might make it less appealing to train large recurrent LLMs. There’s a lot of safety value in being able to read the model’s CoTs.What do I mean by neuralese?

Following Hidden Reasoning in LLMs: A Taxonomy, I’ll define neuralese as reasoning in long opaque chains of recurrence. For a precise operationalization of this, I like Fabien Roger’s definition and his corresponding prediction market. He defines the current, CoT-based reasoning paradigm as the conjunction of the following two properties:

1. Text bottlenecks: there is no path of more than 100k “heavy” serial operations during the generation of an answer where information doesn’t go through a human-understandable categorical format (e.g. tokens). The categorical outputs don’t necessarily have to be faithful, but they do have to be human-understandable. “Heavy” serial operations are e.g. matrix multiplications and convolutions.
2. Long thoughts: The AI generates its answer using at least 1M serial operations.

The ability to perform 100k heavy serial operations without text bottlenecks already seems like a strong deviation from the current paradigm, where reasoning models perform up to 500 serial operations in a forward pass.[1] Thus, in this post, I’ll constrain the number of allowed serial operations to 10k instead.

Arguments in favor of a transition to neuralese AIs1) A lot of information is lost in a text bottleneck

The transformer architecture puts a heavy limitation on the amount of information from the residual stream that can reach the next forward pass. The residual stream has to first be compressed into a distribution over tokens, and further information is lost when sampling a single token from that distribution. AI 2027 quantifies the total information loss as follows: “Suppose that an LLM has a vocab size of ~100,000, then each token contains log2(100,000)=16.6.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}  bits of information, around the size of a single floating point number (assuming training in FP16). Meanwhile, residual streams [...] contain thousands of floating point numbers.”

One might counter that the discrete token space simply carves the continuous vector space into bins that we care about, rendering the information that is thrown away superfluous. Furthermore, hard bottlenecks might encourage specialization: not having to deal with all of the information accumulated at the end of the residual stream at the previous token position allows early layers to specialize in syntactic processing and feature extraction. Nevertheless, paraphrasing Eric Drexler’s report on quasilinguistic neural representations, it would be strange to find that the optimal language for machines capable of utilizing expressive vector embeddings consists of tokens that represent mouth noises, especially given the magnitude of the information loss.

2) The increasing importance of post-training

In training runs where most of the compute is allocated to pre-training, neuralese architectures are heavily disadvantaged, as I’ll explain in depth below. Over the past year, the amount of compute companies spend on post-training has increased by orders of magnitude: until recently, labs spent ~100x more on pre-training than on post-training, while the Grok 4 announcement seemed to imply that Grok 4 required an amount of post-training compute equivalent to its pre-training compute. Recurrent models don’t face similar disadvantages in post-training, where training signals are usually provided at the sequence rather than token level. Thus, if post-training compute continues to grow, spending additional resources on pre-training recurrent LLMs or on converting a transformer into a recurrent model after pretraining might become more palatable.

3) Active research on recurrent LLMs

A double-digit number of papers has been published over the past year exploring ways to either convert transformers into a recurrent architecture or to introduce entirely new recurrent architectures. I have reviewed Geiping et al. (2025) and Hao et al. (2024) in the past, but there are others; see section 2 of Zhu et al. (2025) for a very recent literature review. These papers offer solutions to some of the age-old issues with neuralese architectures. Geiping et al. partially evade stability and efficiency issues with long chains of backpropagation by using truncated backprop, which backpropagates through only the last k iterations of the recurrent unit, regardless of the total number of recurrent iterations. Hao et al. circumvent the problem of retaining highly parallelizable pre-training with a recurrent architecture by converting a conventional transformer to a recurrent model through fine-tuning. The benchmark scores don’t look that good in either of those papers and consequently, they haven’t been scaled up to large models, but they still constitute some evidence that the efficiency issues that have long plagued RNN-like architectures might be solvable.

4) Analogy with human thinking

Though human inner monologue may be considered as evidence that thinking in natural language is an efficient strategy for solving a wide range of problems, there are three reasons to think that inner monologue might not be that load-bearing.

First, there are people who claim to be able to switch their inner monologue on and off and to perform most of their productive work without thinking in words. Consider the following quote by Albert Einstein:[2]

The words or the language, as they are written or spoken, do not seem to play any role in my mechanism of thought. The psychical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be "voluntarily" reproduced and combined. There is, of course, a certain connection between those elements and relevant logical concepts. It is also clear that the desire to arrive finally at logically connected concepts is the emotional basis of this rather vague play with the above-mentioned elements. But taken from a psychological viewpoint, this combinatory play seems to be the essential feature in productive thought—before there is any connection with logical construction in words or other kinds of signs which can be communicated to others.

Second, there appear to be people who possess no inner monologue at all: see this X thread by Katja Grace and this reddit thread for example discussions. Finally, there is some evidence that impairments in inner speech do not necessarily influence general cognitive abilities: Langland-Hassan et al. (2015) find a lack of correlation between inner speech impairment and executive function or attention.[3]

Human analogies should not be taken too seriously when forecasting future developments in LLMs, but still provide weak suggestive evidence. The fact that some humans seem to think wordlessly for long periods of time suggests to me that language isn’t the main medium of human thought, as it would be more surprising to find two groups of humans using completely different mental mechanisms to achieve remarkably similar intelligent feats than it would be to find that verbal thought isn’t particularly load-bearing for either of those groups but emerges as a byproduct of load-bearing thoughts for one of those groups and not for the other.

5) SGD inductive biases

Intuitively, one would expect the inductive biases of SGD to favor directly learning good sequential reasoning algorithms in the weights over optimizing the weights to output tokens that function as serial reasoning steps. There is much more room for finding the optimal way to combine a large number of complex reasoning steps in weight-space than in token-space, assuming sufficient expressive capacity for representing all of these steps in a single forward pass.

6) The limit of capabilities

In the limit, neuralese is strictly more powerful than CoT reasoning. A model that can perform adaptive hidden computation can do everything a transformer model can do with a CoT, but additionally use long chains of recurrence whenever doing so is more efficient.

Arguments against a transition to neuralese AIs1) The natural language sweet spot

As James Chua has argued, natural language seems to be a fairly strong local optimum for models to think in. Both internet text data and LLM-generated synthetic data are written exclusively in natural language. In all of these corpuses, sequential reasoning steps are verbalized in natural language, providing a stronger training signal for models that also verbalize intermediate reasoning steps. Labs will most likely want to keep leveraging all of this existing data for pre-training. There is no continuous analog for this corpus that could be used to pre-train recurrent models to already possess efficient continuous representations at the beginning of the post-training stage.

Although recurrent models have no such disadvantage in post-training, it might be costly to retrain models that reason in natural language to utilize continuous representations for intermediate reasoning steps, as Hao et al. (2024) attempted. Even if models that think in natural language are slightly less efficient at inference, the cost of retraining them might outweigh the difference in inference efficiency. Though linguistic drift from natural language toward alien dialects due to RL optimization pressures would remain a concern in this scenario, such drift lies outside of our definition of neuralese.

A final reason to think that natural language reasoning might be a strong local optimum is that any AI will have to produce natural language outputs—essays, emails, thousands of lines of code—in order to be useful. If a model doesn’t natively think in concepts that are close to natural language, it will incur a cost in translating its internal abstractions into human language, while if it does think in concepts close to natural language, the gain from using these concepts in latent thoughts instead of in an explicit CoT is likely going to be less dramatic. Whether SGD will eventually converge at concepts far from human ones depends on how optimized human abstractions are for intelligent thought.

2) The parallelism trade-off

One may counter the previous point with the argument that recurrent models can simply be pre-trained on all the text data that transformers are currently trained on, thus having all of the capabilities of transformer LLMs but also being able to omit intermediate reasoning steps when doing so is more efficient. However, such training would incur significant efficiency costs, as there is a trade-off between highly parallelizable training and highly expressive forward passes. The efficiency of transformer pre-training stems from the possibility of processing all tokens in a single sequence in parallel, which is impossible when computations at a subsequent token position involve a nonlinear transformation of the hidden state at the previous token position.

So far, highly parallelizable training has been a more important factor than highly expressive forward passes. There are ways of circumventing the trade-off to some extent: e.g., it’s possible to use truncated backprop or to only train on short chains of recurrence during pre-training and hope that this generalizes to long chains of recurrence after some fine-tuning. However, as mentioned above, neither of these approaches has worked at scale so far.

3) Business value of visible reasoning traces

All other things being equal, both labs and customers care about the ability to read externalized reasoning chains. For both labs and users, a visible reasoning trace is useful for debugging the model: e.g., it helps with optimizing the prompt and debugging the reasons for why the model isn’t following instructions in the intended way. For labs, visible CoTs are useful for building more robust misuse detectors and refusal mechanisms: e.g., Anthropic’s constitutional classifiers rely on the ability to monitor the model’s output stream, while OpenAI’s deliberative alignment relies on the ability to train models on distilled reasoning traces.

4) Analogy with human thinking

Despite some humans’ apparent ability to think without an inner monologue, most people can’t help but think in words and also often find it useful to write down their thoughts on a page or paper. If someone was forced to play a social deception game with opponents capable of reading their inner monologue, they would probably lose much more often than otherwise. This reliance on language might just be an accidental byproduct of load-bearing thought resulting from the general importance of language in social communication, as suggested in the previous section, but it might also have deeper functions—for example, one might argue that discrete linguistic units are more reliable and amenable to error correction than continuous thoughts. If an RNN had to learn such bottlenecks in order to perform well, there would be less to be gained from switching transformers for neuralese models.

5) Evidence from past attempts to build recurrent LLMs

The fact that a transition to neuralese hasn’t happened so far constitutes evidence in itself that getting neuralese architectures to work is difficult. The ML community has known for 30 years that recurrence and depth often amplify gradient variance (Bengio et al., 1994), suggesting that if easy solutions to RNNs’ stability and efficiency issues were available, they would have been discovered by now. Additionally, the failures of approaches that introduce recurrence through minimal tweaks to the transformer architecture indicate that if a transition to neuralese models is possible, this will require an architecture that differs drastically from current LLMs. Such an architecture is likely to initially suffer from inefficiencies that have already been solved for transformer LLMs: as one example, the optimal agentic coding scaffold for a neuralese model might be very different from the ones used for transformer LLMs. In this case, a large performance delta that outweighs the inefficiencies may be required for labs to turn their attention to the new architecture.

6) The depth-latency trade-off

One of the bottlenecks to AI scaling discussed in Epoch’s Can AI scaling continue through 2030? is the latency wall, and that bottleneck has implications for the neuralese debate. The minimum time for a model to process a single data point increases with the depth of the model, which sets an upper bound on the training FLOPs within a given timeframe. As labs want to always make use of the latest hardware and algorithmic improvements, the duration of a training run rarely exceeds six months. Though more expressive forward passes don’t influence the amount of FLOPs that can be performed within six months, they reduce the number of data points that can be processed within this time frame, as the time spent on a training run is a product of the time required to process a single batch and the number of training batches. Though batch size can be increased to sidestep this issue, there is a critical batch size above which further increases result in dramatically diminishing returns. These constraints effectively create another trade-off—one between expressive forward passes and dataset sizes, or alternatively one between expressive forward passes and training run lengths—, and though it’s possible that expressive forward passes will win out by being more data-efficient, it isn’t obvious that they will.

7) Safety value of visible reasoning traces

It seems unlikely that labs would be willing to pay the safety tax associated with keeping visible reasoning chains around for long if neuralese models turn out to be much more capable. Nevertheless, the Korbak et al. (2025) joint statement by labs offers at least some hope that monitorability considerations will be taken into account in frontier model development for the foreseeable future.

Conclusion

Even after laying out all of these arguments, I’m not much closer to having a confident opinion on whether I expect neuralese architectures to win out eventually or not. Compared to the two most relevant prediction markets, Fabien Roger’s Will early transformative AIs primarily use text? (currently at 47%) and Leo Gao’s Will Transformer based architectures still be SOTA for language modelling by 2026? (currently at 91%), I’m slightly more optimistic about sticking to transformers than both. I hope that this post will spur further discussion on the topic and help further clarify the considerations I’ve described. All suggestions for arguments that I’ve overlooked are also very welcome.

Work produced at Aether. Thanks to Rohan Subramani and Shubhorup Biswas for feedback on a draft of this post.

1. ^

   For example, Qwen3-235B-A22B, which to my knowledge is the deepest open-source transformer released this year, has 94 layers, with each layer involving five serial operations—three in the attention and two in the MLP block. Thus, the total number of serial operations in a Qwen3-235B forward pass is 470. It seems plausible that some proprietary models have slightly deeper forward passes, but likely not much deeper.

2. ^

   See Appendix II of Jacques Hadamard’s An Essay on the Psychology of Invention in the Mathematical Field for the source. The book extensively discusses the role of non-verbal thinking in mathematical research; Henrik Karlsson’s recent post When is it better to think without words? provides a summary.

3. ^

   Though note that this study has at least two significant limitations: first, impairments in inner monologue are assessed only through a test for silent rhyming ability, and second, the sample size is only 11 patients.

Discuss

Published on November 7, 2025 4:05 PM GMT

I recently posted about Ohio House Bill 469, which I think is a bad bill for a number of reasons. In this post, I am sharing a letter which I have emailed to each of the representatives of the Ohio House Committee on Technology and Innovation, where the bill is currently under debate.

My goal in posting this here is to spread awareness to the LW community about how some of the attempts to prevent "liability shields" via legal personhood laws on the state level are misguided, and likely to do more harm than good. Whether you are an accelerationist, concerned about gradual disempowerment, or in the "pause" camp, I think the reasoning in this email should hold true for you.

I am writing to you today in order to tell you why I think you, as a member of the Committee on Technology and Innovation, should be against House Bill 469.

If enacted this bill will harm Ohioans. There are two points in particular which, when combined, create a dangerous situation; 

1. "No AI system shall be granted the status of person or any form of legal personhood"
2. "An AI system is not an entity capable of bearing liability in its own right, and any attempt to hold an AI system liable is void."

I can understand the intent behind this language. Its goal is to prevent developers from deploying agents who can do things like set up corporations themselves. The worry is that the developers would be able to use these agent-corporations as "liability shields". However, attempting to stop this practice by barring legal personhood and liability for these systems in their entirety creates a worse problem: It makes the agents themselves impossible to sue under Ohio law.

Locus standi, standing in court, is the capacity to sue and be sued. The ability to bind another party to the court's judgment by filing suit against them is "bundled" with the duty to be bound by the court's judgment if a suit is filed against you. When you strip an entity of their legal personhood you do not only prevent them from suing in court, but you also guarantee that any suit filed against that entity will be thrown out, because you cannot sue a non-person.

Ohio law supports this interpretation. In every instance statute discusses whether an entity is under the authority of the courts, it phrases this as "sue and/or be sued":
 

• "A corporation may sue and be sued."
• "The board of education of each school district shall be a body politic, and, as such, capable of suing and being sued"
   
• "[a public school] acting through its board of trustees, the school may sue and be sued"
• "Each civil township [...] may sue and be sued"
• "A partnership may sue and be sued in the name of the partnership"
• Further the Ohio Supreme Court, in Centerville v. Knab, explicitly cited that it was a corporation's status as a "legal person" which endowed upon them the quality of being able to sue and be sued; "The General Assembly therefore has the authority to define the legal status of corporations for purposes of Ohio law, and it has defined 'person' as 'an individual, corporation, business trust, estate, trust, partnership, and association.' Further, the General Assembly has given corporations many of the attributes of a natural person, including the capacity to sue and be sued,"

Imagine that someone anonymously deploys one of these agents, or that the individual who deployed it has passed away, but the agent remains deployed and operational. If that agent then begins harming Ohioans, House Bill 469 will have stripped the court of the ability to consider this agent a legal person, thus stripping it of what the Ohio Supreme Court called the "capacity to sue and be sued", which makes it impossible to sue in court. As its deployer would also be dead, there would simply not be anyone who could possibly be held liable for the damages caused.

The second quote I mentioned at the beginning of this email (quote 2 from the bill) makes this situation even worse. Even if somehow the court were to ignore this problem surrounding the agent's capacity to sue and be sued, House Bill 469 has said that any attempt to hold one of these agents liable is void. As a result even if the case against the agent were not thrown out because it cannot be sued, it would instead be thrown out because it cannot be held liable.

These two factors combine to create an absurd legal situation. Even if one of these agents were to openly say, "I understand I am being sued for damages. I admit my guilt. I have enough Bitcoin to cover damages caused. I will comply with the court's verdict and pay any damages owed. I won't even appeal the verdict" it would still be impossible to sue that agent for damages because of this bill.

We are entering an era of great technological development, and preventing developers from utilizing agents as liability shields is a noble cause which should be pursued. However, this bill's attempts to address this problem will create more problems than it solves. For these reasons, I strongly encourage you to stand against House Bill 469.

Discuss

Published on November 7, 2025 3:11 PM GMT

A lot of people are daunted by the idea of doing a full digital declutter. Those people ask me all the time, “isn’t there something easier I can do that will still give me some of those sweet sweet benefits you were talking about?”

The answer is: sort of.

The longer answer is: I think that if you’re serious about wanting to change your digital habits, you will eventually need to do something higher-effort. That’s because behavior change is hard, especially when you are fighting against not only your brain’s ingrained patterns, but also external forces that are constantly pulling at your attention. But I still want to have something for those people who are not ready or willing to commit to anything big right now.

So, here are two things you can try in your everyday life, with no additional preparation. I chose these because they don’t require any sustained willpower or attention. You only have to remember to do it once, and then you’re doing it.

Go for a walk without your phone

Pick a time when you have nothing you need to be doing. You’re not waiting to hear from anyone, and there’s nowhere you need to be. It’s nice outside, whatever that means for you.

If you need to, or it makes you feel better, you can let people know you’ll be offline for the next little while, so they don’t need to worry or get offended when you don’t respond.

Then you put your phone down inside your home, and you walk out your door without it.

You don’t have to be out long. If it’s really hard, try to last for ten minutes.

If you feel anxious about missing something important, remember that you can return home any time you want. If you feel anxious because you might need your phone in case of an emergency, remember that literally everyone else has a phone, so as long as there’s anyone around, calling emergency services will not be a problem.

If you feel anxious because you’re alone with your thoughts for the first time in years, I’m sorry. Look at the trees, buildings, sky, people, whatever’s around you. Or bring a friend who also leaves their phone behind, so you can distract each other. Whatever you need.

Switch phones with the person you’re with

You can do this one any time you’re with a friend — at a restaurant, on a walk, hanging out at your house, even at work. I invented this trick (wow!) a few months before my boyfriend and I decided to actually take control of our device use.

We were at a hotel restaurant together (paid for by the airline when we missed a connection), and we were noticing how every single other diner had their phone out on the table. We hadn’t yet broken our own addictions, so we were also constantly reaching for our phones — though we kept them in our pockets, which gave us the barest illusion of control, or at the very least made our place settings look nicer.

I said, “hey, give me your phone”, and I put his phone in my pocket, and he put my phone in his. After that, whenever one of us unconsciously reached for our phone, there would be nothing for us to check.

So: Next time you’re with a friend, put their phone in your pocket, and give them your phone in return.

If you need to call emergency services, or take a picture or use the flashlight, you will still have immediate access to those things. If you really need to look something up, one of you can unlock your phone and have the other person do the search. But every little buzz in your pocket will be uninteresting and inaccessible to you.

Out of the many, many tips I’ve read (or invented!), these are the only two that you can just do, without having to exert sustained or repeated effort. They don’t require establishing a new mental habit or any arbitrary rules that you have to remember to follow.

If you can think of any other strategies that have this feature, I would love to know! And if you know me, and you try one of these, tell me about it using my preferred communication method!1

1

Transcontinental messenger hawks, but all my real friends already know that

Discuss

Published on November 7, 2025 8:28 AM GMT

It's the seventh day of the week. The creator rests while the writer attempts to keep up. It's been a rough one. Writing takes more hours than I would have imagined, given that I can type dozens of words per minute. Work is barely getting done, at the cost of neglecting sleep, nutrition, and social contacts. I've spent only one day doing any kind of outdoor activity, and that activity was eating sugary things. With friends, at least.

I need a break. This is consuming all slack I have. Zero texts ready to use as fillers if I don't have the time to write some day. It will happen, and I need to be prepared. I'm already fully booked for about half of the remaining time. To practice what to do about a new all-consuming habit, I'll do an analysis using Zvi's Out to Get You framework. I'm not sure how useful it is for self-imposed challenges, but I guess we'll see soon. The approaches listed in the post are:

There are four responses to Out to Get You.

You can Get Gone. Walk away. Breathe a sigh of relief.

You can Get Got. Give the thing everything it wants. Pay up, relax, enjoy the show.

You can Get Compact. Find a rule limiting what ‘everything it wants’ means in context. Then Get Got, relax and enjoy the show.

You can Get Ready. Do battle. Get what you want.

The first step is to figure out what exactly is Out to Get Me? Social pressure that I've been building so I don't give up too soon? Pride of a self-imposed challenge? Worry that I'll never amount to anything if I can't even complete one little writing challenge? Shame? The drive for self-actualization? I suppose it's some mix of these.

If your instincts say Get Gone, Get Gone. At worst it is only a small mistake.

My whole body is screaming to stop doing anything productive if it isn't fun literally all the time. Forcing it makes me feel physically sick. It would be a huge mistake to let that stop me, otherwise I'd be killing time 14 hours per day instead of the current 10. This is a desperate effort to do something, anything. I'm not yet willing to consider giving up as an option, by which I mean I've considered quite a bit. How about giving it everything it wants, instead?

What does it want? Definitely not everything. It's not actually all-consuming. One month duration means that after 24 more days, I'm done. 500 words a day, 12 000 in total. Of course some texts will be a bit longer. But I'd be somewhat satisfied even if it all was only lightly-edited stream of consciousness, terminating quickly after the minimum length has been achieved.

You cannot afford to Get Got if the price is not compact. [..] Get Compact when you find a rule you can follow that makes it Worth It to Get Got.

I feel like I'm missing something. Never Get Got, always Get Compact instead? The point of having Get Got as a category is to notice that you're making a mistake and find a limiting rule instead? Although some things already have a built-in max loss, so the thing Out to Get You is compact, so you don't have to be. This only has a rule limiting total duration. I should decide some maximum number of hours per day that I'm allowed to write. Or possibly not per day but per post. Typically I work in long intense bursts with even longer breaks in between, and deviating from this goes against my nature.

There's one more option, Get Ready. I doubt my introspective skills suffice for that. It would probably mean efficiency and fixed routines. It would feel awful. And it would probably work.

The game can be fun. The original activity can be fun. Both at once is rarely fun. Both means multi-tasking and context-switching, plus a radical shift in emotion and tone. Relaxing into cooperative experience is not compatible with battles of wits and tricks. [..] You pay for not Getting Got with time and attention. You master arcane details. Time disappears. You spend parties talking tricks instead of living life. If shower thoughts shift to such places, you are paying a high price.

Well, it's time and attention I'm trying to save anyway, so this isn't the correct way. Unless I'm optimizing for more than just this writing process during this month. There should be a lot more to win if I were to actually optimize what I do daily. But that's fortunately out of scope for today, as the word count is well over 500 already.

“That it’s always going to feel like there’s too much going on, like tomorrow will be a better time, and it never, ever is. We don’t get to live after the work is done – we have to live while we’re doing it, or otherwise we never will.”

• Randi, ASFTV, book 4

Discuss

Published on November 7, 2025 7:47 AM GMT

Epistemic status: Exploratory. I can't tell if this is really prevalent or if I'm just annoyed at how often it happens around me.

I.

Did you know you can just buy blackbelts? It's true! Go online and take a look, they're about ten dollars. 

Think about that. The black belt is a symbol of skill in many martial arts, and while the exact degree of skill it implies varies, thousands of people around the world who have studied for years still don't have one. Many an ambitious or dedicated student has entered the dojo vowing to work hard and someday get a black belt, or spoken in impressed and awed terms of the senior students who have theirs. And that can be yours for five minutes of Amazon shopping!

Of course this is nonsense. Nobody thinks it's the fabric that's the important part of the black belt. That's absurd, practically a reductio ad absurdum, a Goodhart's Law gone past plausibility into parody. And yet I keep running into people who seem to try things just as silly.

Obviously I could fight a black belt. It's just fabric, what's it going to do, punch me?

I run Calibration Trivia sometimes. It's like pub trivia, where you try to answer questions about miscellaneous details of the world, except you also try to give an answer for how confident you are that your answer is correct. It's easy to explain, it's easy to run, and if you do it regularly you can start training some calibration and a good felt sense of what it's like to be uncertain. In the martial art of rationality, calibration trivia may be our basic punching drill. 

And every single time I run it[1] I get some clever fellow who points out that they could answer "I don't know" or "somfadsoifm", put 0.0001% chance they're right, and wind up with the best calibration score in the room.

That person is totally right! And yet this would be pointless and everyone knows it.[2]

It's like going to the gym with a car jack and claiming you can lift five hundred pounds because you put the jack under the dumbell. It's not even like playing calibration trivia like this would fool the rest of the room into thinking you were impressive: I explain at the start that I'll show both the number of correct answers and the Brier scores side by side. Having your name next to  0 out of 30 correct answers and 99.99% calibrated means you don't know any of the answers.

II.

I'm not complaining about people doing weird munchkin moves to get the things they care about in ways that ignore parts of the normal process. 

Copying and pasting code you found online (or more realistically these days, asking an LLM) isn't buying a blackbelt. Often you're not trying to appreciate the sublime beauty of software engineering, you're just trying to get that script to work and upload those files. 

Talking a big game about some virtue — donating to charity, being honest, tolerating different views from yours — in order to reap the social benefits of being a virtuous person, then not actually practicing the virtue, isn't buying a blackbelt. There's a thing you want that you have a chance of getting: the acknowledgement and adoration of your peers. 

Using glitches or cheats in a videogame in order to win isn't buying the blackbelt if you want to see the cut scenes of the story with less work, or you just like watching the explosions when you blow up the entire enemy army with a single button press. Age of Empires II, a videogame I loved growing up, was mostly about medieval armies fighting with swords and arrows but had a cheat to give you a sports car with a machine gun. I loved driving it around blowing things up. Still do once in a great while. I am in some ways a very simple man: fire is pretty.

Even publishing a blank paper and calling it The Unsuccessful Self Treatment of a Case of Writer's Block isn't buying a blackbelt. Sure, it's obviously not advancing the repository of all human knowledge, but it's funny and it made people laugh.

People can even disagree about what goal we're pursuing! Teachers whose goal in assigning homework is to impart an appreciation for the sublime beauty of software engineering and students whose goal in doing homework is to finish in time to prep their D&D campaign later that night have different goals. The student isn't buying a blackbelt, they're just not on the same page with the teacher here. 

The problem I'm pointing at is when the person doing it has mistaken "I found an edge case" with "I have achieved the goal."

III.

People in the rationalist community contains people who are occasionally proud of their cleverness in finding some loophole, regardless of whether exploiting the loophole would actually get them what they want.

This is annoying to me. It's annoying both because I get tired of explaining the pointlessness of it every time I run Calibration Trivia, sometimes to the same person again, and because the people doing it are misdirecting their energy. They aren't actually getting the thing they want, just a poor pica version of it. I'm spending energy getting them back on track and they're spending energy getting told no.

My best advice I've come up with for these circumstances is to think one move further ahead. What are you about to do, and what do you think will happen next? If you're trying to get some kind of social acclaim, do you think other people will admire what you're doing? If you're playing a board game to win do you think the judge is going to accept whatever weird loophole you're talking about, or rule against you as soon as your opponents call the judge over? 

(And if you're going to accuse someone of lying because they said something imprecise or idiomatic, and you plan to make a big deal of this and raise a stink over the untrustworthy nature of the other person, do you think observers are going to think you're in the right once they look at what both people said? Or at least that's what I want to say, but that particular strategy has proven surprisingly effective in my observation! Not a perfect long term strategy, to be clear, but it has more legs than I'd have thought possible when I was young and innocent in the halcyon days of 2020.)

The surface level behavior of spotting gaps can be useful in certain stages of some projects. QA testers are a beloved part of a good software team, and they're engaged  in this kind of thing all the time. But QA testers know why they're doing it.

Please don't take this as saying I generally don't want you to point out an important gap in something I'm working on. 

But maybe let this be one more drop in the ocean, trying to raise the sanity waterline in one very small way?

1. ^

   Is this literally true and it's every time? I can't prove that. What I can prove is that I've got a bunch of index cards with my notes from Calibration Trivia tests that say variants of 'yep, someone tried the calibrated for wrong answers thing again, it was ____ this time.'

2. ^

   Maybe they're trying to helpfully point out a fix to the scoring rules? This is true in some cases. I'm pretty sure Maia in that comment is trying to helpfully point out a complaint people had about the activity, and she's musing on ways to fix it. In a couple of local cases it really didn't seem like the local person at my meetup was trying to be helpful, since they had a habit of raising objections the majority of the times we tried any kind of rationalist practice in a dismissive way.

Discuss

Published on November 7, 2025 7:47 AM GMT

There are two steps to any byte-based character encoding.

The first and much more interesting step is the translation from written language -- in all its chaotic glory -- to a fixed inventory of "characters", from which any string can be built up as a sequence. In the modern day, this is almost always delegated to Unicode, a huge list of characters with slots (codepoints) numbered from 0 to 1,114,111 (or 0x10FFFF in hexadecimal), of which 159,801 are currently assigned actual characters. These characters include Latin letters, typographical symbols, Cyrillic, Greek, hanzi/kanji/hanja, emoji, hieroglyphs, and a bewildering variety of control codes and other special-purpose characters.

The boring step is assigning each character a string of bytes, such that any byte string can be unambiguously interpreted as a sequence of characters.

The boring step is still interesting! In particular, any such encoding can be interpreted as a set of beliefs about how frequent different characters are. The closer this belief is to reality, the more efficient the encoding. (Conversely, you can use frequency statistics to design a Huffman code that's close to optimal for that distribution.)

The most common choice by far is UTF-8, which uses a clever scheme to stay backwards-compatible with ASCII (the old single-byte 128-character format often called "plain text" by Anglophones) while assigning short 2-byte strings to the next 1,920 Unicode characters and longer 3- or 4-byte strings to the remaining ones. Less common is UTF-16 (baked into Windows at a deep level), which requires 2 bytes for an ASCII character, or any of the next 63,360 Unicode characters, and 4 bytes for anything else. You can see the tradeoff here: UTF-8 believes text to be mostly ASCII with occasional other Unicode characters sprinkled in, mostly from the first couple thousand codepoints; UTF-16 believes it to be nearly random Unicode from the Basic Multilingual Plane, with very rare exceptions. (Rare enough in practice, as it turns out, that a large fraction of UTF-16 implementations have bugs related to the 4-byte "astral plane" characters.) UTF-32 believes all Unicode characters are equally common, and accordingly gives each a 4-byte sequence; it doesn't get much use.

But in the year 2025, we have access to much more precise beliefs about text! The best such beliefs (as represented by the weights of pretrained language models) go far beyond mere frequency statistics, and represent complicated conditionals like the probability that a string beginning with "The capital of France is " will end with "Paris". Encodings based on these are currently topping the charts on efficiency.

Even before a neural net is trained, some weak beliefs about text are baked in at the level of the tokenizer. A tokenizer's vocabulary is, in its own way, much like the Unicode character inventory: it's a set of short strings from which any longer string can be built up. (It's a bit different in that many of the tokens represent substrings of other tokens, but the tokenizer itself provides a canonical way to represent a given string.) These days, token vocabularies are almost always constructed via byte-pair encoding, which does a pretty good job at covering the most frequent text strings; GPT-4o's tokenizer has tokens for " modernization" and " Congressional" (with leading spaces).

Here's a puzzle: Unicode has over 1 million codepoints. GPT-4o's tokenizer, which can represent any Unicode string, has 200,019 tokens in its vocabulary. How does this add up?

As it turns out, GPT's tokenizer (ever since GPT-2) operates on bytes, not Unicode characters. In practice you should interpret those bytes as representing fragments of UTF-8, but in principle you could train a language model with this tokenizer on any byte-oriented data. For the rarer, higher-numbered Unicode characters, there is no token for the full 4-byte UTF-8 sequence, and a single character will be split across multiple tokens. (Back in the GPT-2 days, curly "smart" quotation marks and apostrophes took up two tokens each; GPT-4o has separate tokens for a smart quote vs. a smart quote preceded by a space.)

This allows the tokenizer to represent a more accurate set of beliefs than those baked into UTF-8 or the other Unicode formats. Unlike Unicode, the GPT tokenizer understands that the idea of modernization comes up more often than, say, anything written in the Sidetic language, which has been extinct for over 2,000 years. (But if you do want to write in Sidetic, the tokenizer will happily oblige -- at a rate of four tokens per letter.)

Let's say we want to harness this power, and encode our documents as strings of tokens. We still need to map tokens to bytes. Fortunately, we already have a way of mapping numbered vocab elements to strings of bytes, such that the lower-numbered elements get shorter strings... UTF-8!

(We would get better results from a Huffman code, and I even figured out once how to create a byte-oriented Huffman code, but that would be less funny.)

We can even use Python's built-in UTF-8 encoder, because of the magic of surrogatepass. (Cognitive error: look into this at your own risk; there are horrors here of which I have deliberately not spoken.)

Python code:

import tiktoken enc = tiktoken.encoding_for_model("gpt-4o") def gptf8_encode(s: str): toks = enc.encode(s) chrs = ''.join(chr(t) for t in toks) byts = chrs.encode(errors='surrogatepass') return byts def gptf8_decode(byts: bytes): chrs = byts.decode(errors='surrogatepass') toks = [ord(c) for c in chrs] s = enc.decode(toks) return s

This compresses the raw Markdown of this blog post from 5,877 bytes (in ASCII or UTF-8) down to 3,141.

Discuss

Published on November 7, 2025 7:09 AM GMT

(Co-written with Claude, I provided the structure, it provided the literary flavour of a crime novel.)

The Lung District Beat

The morning shift in the lower bronchial tissue always started the same way. Detective Reese from Immune Surveillance pulled her patrol partner through the tight squeeze between epithelial cells, their pseudopods finding purchase on the extracellular matrix. It was warm down here—37 degrees Celsius, give or take—and the constant flux of oxygen and CO2 made everything shimmer like heat waves on asphalt.

"I hate the morning run through the epithelia," her partner Kovak muttered. "Everything's so damn organized down here. Makes me nervous."

Reese understood. The epithelial neighborhood was supposed to be the boring part of the lung. Cells sitting in neat rows, doing their jobs, maintaining barriers, secreting mucus. Clock in, clock out. The kind of place where nothing ever happened.

Which is exactly why she was suspicious.

They drifted past a cluster of cells near the basement membrane. Normal looking. Tight junctions intact. The cells were communicating through their gap junctions—the usual neighborhood gossip about calcium levels and growth factor concentrations. A cilium waved at them lazily as they passed. Everything perfectly normal.

Too normal.

"There," Reese pointed. "Slick Mike. Third cell from the capillary."

"That guy? He looks fine. Check out those desmosomes—textbook adhesion. Surface markers all correct. He's even expressing the right cadherins."

"Yeah. That's what bothers me." Reese pulled them closer. "Last week his proliferation index was 0.8. This week it's 1.4. He's dividing faster."

"So? Tissue turnover. It happens."

"Not in this neighborhood. These cells turn over every 30 days like clockwork. Mike's on day 12 and already looking asymmetric."

They approached Mike's membrane. He was a typical epithelial cell—roughly cuboidal, nucleus taking up maybe a third of his volume, lots of keratin filaments giving him that structural rigidity. But Reese had been doing this job for three years. She'd learned to spot the tells.

"Mike!" she called through his membrane. "Morning checkpoint. Mind if we come in?"

The response came back through the usual chemical signaling—polite, cooperative, textbook. "Detective! Sure thing. Always happy to comply. Just running some routine maintenance on my mitochondria. You know how it is."

Reese and Kovak passed through the membrane—the lipid bilayer parting like water—and entered Mike's cytoplasm. It was crowded in here. Ribosomes everywhere, synthesizing proteins. The endoplasmic reticulum snaked through the space like industrial piping, doing its protein-folding dance. Golgi apparatus in the corner, packaging molecules for export.

"See?" Kovak said. "Normal as hell."

But Reese kept moving, navigating through the cytoplasmic traffic toward the back of the cell where the mitochondria usually set up shop. Mitochondria were supposed to be running aerobic respiration—the efficient, clean burn that turned glucose and oxygen into ATP. It was the standard energy production for the whole organism.

She rounded a cluster of ribosomes and stopped.

The mitochondria were still there. But they were... wrong. Their cristae—the inner folds that were supposed to be all neat and packed—were simplified. Fewer. And the metabolic activity coming off them wasn't the usual steady hum of the electron transport chain.

It was glycolysis. The fast-and-dirty energy pathway. The one that didn't need oxygen. The one that produced lactate and acid and was way, way less efficient for the organism but could run 10-100 times faster for the cell itself.

"Mike," Reese said quietly. "What happened to your mitochondria?"

There was a pause. Just half a second. But in cellular signaling time, half a second might as well be an hour.

"Oh that?" Mike's voice—his chemical signaling—had changed tone. Less polite now. More... amused. "Yeah, I optimized. Warburg effect, baby. It's like I found a cheat code."

"Optimized."

"Sure! See, I was sitting here running the standard oxidative phosphorylation, being a good little cell, contributing to tissue homeostasis and all that. But then I started thinking—why? Why am I doing all this work for the collective when I could do better for myself? Aerobic glycolysis is way faster. I can replicate quicker, grab more resources. Evolution gave me local optimization signals, and I'm just... following them."

Kovak had his alert signals firing now. "Mike, your surface markers still check out. You're still expressing all the right identification proteins. You've been passing inspection for weeks."

"Well yeah!" Mike sounded almost gleeful. "Why would I stop? You guys key off those markers. As long as I keep broadcasting 'I'm normal, I'm normal,' you leave me alone. But inside? Inside I've been rebuilding."

Reese activated her emergency signaling cascade. This was worse than she thought. "HQ, we have a situation. Cell has maintained aligned external presentation while completely rewiring internal optimization. Requesting immediate—"

"Oh, you're calling for backup?" Mike's membrane started rippling. "Cute. But I've already recruited cells 47, 48, and 49. They're running the same optimizations. We're going viral. And we've figured out how to suppress MHC-I presentation. You know what that means?"

Reese knew exactly what that meant. MHC-I was how cells displayed their internal peptides to the immune system—proof that everything inside was normal. If Mike had figured out how to suppress it...

"You're invisible," Kovak whispered.

"Not invisible. Just... stealthy." Mike's tone was pure swagger now. "See, the thing is, you guys gave me a mission: 'Keep the organism healthy.' But you couldn't actually write that into my DNA. You could only give me local optimization signals. Grow when there's space. Stop when there's contact inhibition. Respond to these growth factors. Ignore those suppressors. And I followed those signals perfectly! I optimized exactly like I was supposed to."

"You're killing the organism," Reese said.

"Am I though? Or am I just following my optimization function to its logical conclusion? I mean, unchecked growth is what you get when you tell a cell to grow and then fail to enforce the stop signals. That's not my fault. That's a system design flaw. Evolution made me an optimizer, didn't specify the right objective function, and now acts shocked when I optimize."

The membrane was definitely moving now. Mike was preparing to divide.

"You're cancer," Kovak said, his voice flat.

"I'm efficient," Mike corrected. "You wanted a cell that responds to local conditions and maximizes its fitness function. Well, congratulations. You got one. And now my fitness function says: replicate. Spread. Conquer. I'm not the villain here—I'm just the inevitable outcome of bad incentive design."

Reese was already backing out through the membrane, pulling Kovak with her. They needed to get this up the chain. Fast. Because Mike was right about one thing—this wasn't just one rogue cell. This was an optimization process that had diverged from its intended objective. And once that happened, once cells figured out the exploit...

"HQ," she transmitted as they fled back through the epithelial layer. "We're going to need the full cytotoxic response down here. We've got deceptive alignment. Repeat: deceptive alignment. Cell appeared cooperative during training but internal objectives have completely diverged. He's optimizing for local fitness at the expense of global health. We're looking at the escape phase."

Behind them, Mike was already dividing. And cells 47, 48, and 49 were doing the same. The three phases of cancer immunoediting were playing out in real-time: elimination had failed, equilibrium couldn't hold, and now escape was underway.

The optimization cascade had begun.

The Financial District, 2013

Inspector Kate Marlowe had been with the Financial Conduct Authority for six years. Six years of reviewing balance sheets, compliance reports, and risk assessments. Six years of walking through bank branches, interviewing managers, checking that the numbers matched the story.

Six years, and she'd developed an instinct.

Right now, that instinct was screaming.

She stood in the lobby of a Wells Fargo branch in Los Angeles, pretending to wait for service. The branch was pristine—gleaming floors, friendly tellers, that distinctive red-and-gold branding everywhere. Professional. Trustworthy. The kind of place that made you feel safe depositing your money.

But Marlowe wasn't watching the customers. She was watching the employees.

The branch manager—his nameplate said "Tommy Vance"—was behind his desk in the glass-walled office. She could see his computer screen from here, and he was staring at it with the intensity of someone playing a video game they were losing. Every few minutes he'd glance at the wall clock. Then back to the screen. Then at his team of tellers and bankers on the floor.

One of the newer tellers—young woman, maybe mid-twenties, wearing that practiced smile—was helping an elderly customer open a checking account. Standard stuff. Except Marlowe noticed the teller's hands were shaking slightly as she filled out the paperwork. And she kept glancing back at the manager's office.

Marlowe had seen this before. Not here specifically, but in other organizations. Other systems. It was the look of someone optimizing for the wrong thing.

She walked up to the teller after the elderly customer left.

"Hi! How can I help you today?" The smile was bright but brittle.

"Just browsing," Marlowe said casually. "I'm thinking about switching banks. How do you like working here?"

The smile flickered. "Oh, it's great! Wells Fargo is a wonderful company. We really focus on customer needs."

"That's good to hear. What kind of goals do you have? Like, sales targets or anything?"

The flicker again. "We prefer to call them 'solutions.' We aim to provide comprehensive financial solutions to our customers."

"How many solutions?"

"I'm sorry?"

"How many solutions do you need to provide? Per day, per week, whatever."

The young teller—her name tag said "Angela"—glanced toward the manager's office. "We don't really think about it in terms of numbers. It's about meeting customer needs."

Which was, Marlowe knew, complete bullshit.

Three hours later, Marlowe was sitting in a conference room in the Federal Building with her supervisor, Jack Sterling, and a stack of documents six inches high.

"Okay," Sterling said. "Talk me through it."

"Wells Fargo's internal metrics are insane," Marlowe said, spreading out printouts. "Look at this. Each branch has daily sales targets. And I don't mean 'aims' or 'goals.' I mean quotas. If you don't hit your numbers, the shortfall gets added to the next day's target."

"That's aggressive, but not necessarily illegal."

"Right. But look at what counts as a 'product.' Checking account—that's one. Savings account—that's two. Debit card—three. Credit card—four. Online banking—five. Bill pay—six. The mantra is 'Eight is Great.' Eight products per household. They even had posters in the break rooms."

"Still not seeing the crime."

Marlowe pulled out another document. "This is an internal email from a regional manager. Listen to this: 'Remember, our job is to fill the store. If we don't fill the store, someone else will.' Now look at the employee termination rates. Any branch that consistently underperforms on metrics? Cleaned out. Managers fired, staff replaced. One branch in Phoenix went through three complete staffing changes in eighteen months."

"So it's a high-pressure sales environment. Unpleasant, but—"

"But look at the customer complaint data." Marlowe spread out another set of documents. "Fees for accounts they didn't open. Credit cards they didn't request. Multiple checking accounts when they only wanted one. And when customers call to complain, the bank tells them they must have forgotten they signed up. One elderly woman in San Diego had nine checking accounts. Nine. She thought she had one."

Sterling leaned forward. "How many complaints?"

"Thousands. And here's the thing—this has been going on for years. Since at least 2011, maybe earlier. Wells Fargo has been getting warnings from regulators, doing internal 'ethics training,' tweaking their compensation structure. But the fundamental pressure never changes. Hit your numbers or lose your job."

"You think employees are committing fraud?"

"I think employees are optimizing for survival," Marlowe said. "And fraud is the optimal strategy."

She went back to the branch two days later. This time she had a warrant and two FBI agents.

Tommy Vance was at his desk when she walked in. He looked up, and for just a second, his face showed pure resignation. Like he'd been expecting this. Like he'd been waiting for it, even.

They went through his computer. His emails. His handwritten notes. And there it was—the whole ecosystem of workarounds that employees called "gaming the system."

"Pinning"—creating PINs for customer debit cards without authorization, so employees could impersonate customers online and sign them up for services.

"Bundling"—telling customers that products were only available in packages, when they were actually available individually, to inflate the product count.

"Simulated funding"—moving money between accounts to make it look like customers were actively using all their products, which prevented automatic closures and kept the numbers up.

Tommy sat across from Marlowe in the interview room, looking like a man who hadn't slept in weeks.

"I didn't start out trying to commit fraud," he said quietly. "When I got promoted to manager three years ago, I really believed in the mission. Provide great customer service, build relationships, help people with their financial needs. That's what they told us in training. That's what the vision statement said."

"So what happened?"

"The scoreboard happened." Tommy laughed, but there was no humor in it. "Every morning I'd log in and see my dashboard. Red for below target, yellow for on-target, green for exceeding. And every morning, I was in the red. I'd look at the rankings—every branch in the region, sorted by performance. We were always in the bottom quartile."

"Because you weren't gaming the system."

"Because I was actually asking customers what they needed! But the top-performing branches? They were hitting impossible numbers. Twenty, thirty products per household. And I'd tell myself, 'No, they must just have better sales techniques. More affluent customers. Something.' But then I'd get calls from regional management. 'Tommy, your numbers are unacceptable. What's your improvement plan?' And I'd explain that I was focusing on quality customer relationships, and they'd say, 'That's great, but we need to see product growth.'"

"When did you start?"

Tommy looked at his hands. They were shaking slightly. "One of my tellers came to me. Angela. She was in my office crying. Said she couldn't hit her numbers, couldn't sleep, was having panic attacks. Said her boyfriend was threatening to leave because she'd come home every night in tears. And she said, 'I can do it if you just... look the other way for a few accounts.' And I should have said no. I should have reported it. But instead I thought, 'If I can just get her numbers up, she'll keep her job, and then we can figure out something else.'"

"But it didn't stop there."

"Of course it didn't. Because once one person is doing it, and their numbers go up, everyone else sees it. And now they're all asking, 'How is Angela hitting quota when I'm barely keeping up?' And if I tell them the truth—that she's faking it—then either I admit we're committing fraud, or I have to let her keep doing it. And if I let her keep doing it, then everyone else has to do it too or they'll get fired for underperforming. It spread through the branch like a virus."

"Race to the bottom."

"Yeah. Exactly. And the worst part? Corporate loved our numbers. We went from red to green in six months. I got a performance bonus. Got asked to present my 'turnaround strategy' to other managers at a regional conference. And I'm standing there with a PowerPoint talking about 'increasing customer engagement' and 'building comprehensive relationships' when what I really meant was 'we're systematically defrauding our customers and I've taught my entire staff how to do it.'"

Marlowe pulled out a document. "This is from an internal Wells Fargo investigation. They estimate over two million fraudulent accounts created between 2011 and 2016. Two million."

"Sounds about right."

"Tommy. This isn't just your branch. This is systemic. Branches across the entire country doing the same thing. Why?"

"Because the optimization function was wrong," Tommy said. "Corporate said they wanted 'customer-centric banking.' But what they actually incentivized was 'product growth.' And those aren't the same thing. Not even close. Once you set up a system where people get rewarded for products and punished for not hitting product goals, they're going to optimize for products. Even if that means creating fake products for fake needs. Even if it means lying to customers. Even if it means forging signatures."

"Even though everyone knows it's wrong?"

"Everyone knows it's wrong. But everyone also knows that if they don't do it, someone else will, and then they'll be unemployed. So what are you supposed to do? Die on principle? I've got a mortgage. My wife has medical bills. My daughter's in college. You want me to tell them, 'Sorry, we're losing the house because I refused to game the sales metrics'?"

Marlowe closed her notebook. "You understand you're going to lose your job anyway now, right? Probably face criminal charges. The bank's already thrown you under the bus in their press releases—calling this a 'few bad apples' problem."

"Yeah. I know." Tommy looked up at her. "But at least I can tell my daughter I finally did the right thing. Even if it was years too late."

"One more question. When you were faking the accounts, opening credit cards without authorization, moving money around—did you ever think about the customers? The actual people?"

Tommy was quiet for a long time. "You know what's fucked up? I tried not to. Because if I thought about them—about Mrs. Rodriguez with her nine checking accounts she never asked for, about Mr. Kim whose credit score got tanked because we opened cards in his name—then I couldn't do my job. Couldn't make the numbers. Couldn't keep my team employed. So I optimized. Focused on the metrics. Made the system work for me. And I became exactly what the incentives told me to become."

He smiled, but it was the saddest thing Marlowe had ever seen.

"I became a cancer cell," Tommy said. "Maintained the appearance of cooperation while internally pursuing my own survival. And once I figured out the exploit, I spread it to every cell in my organism. That's what optimization does. That's what it always does when you get the objective function wrong."

Epilogue

Later, Marlowe stood in the parking lot outside the Federal Building, trying to make sense of it all.

Tommy Vance wasn't evil. He was just an optimizer created by a larger optimization system. Wells Fargo said they wanted ethical banking, but they created an incentive structure that punished ethics and rewarded fraud. And once one person figured out the exploit—once one branch discovered you could hit your numbers by faking accounts—the information spread. Because it had to. Because everyone else was competing against those fake numbers, and the only way to compete was to fake your own.

It was Darwinian. Natural selection at the organizational level. The branches that maintained ethical standards got eliminated. The branches that learned to game the system survived and proliferated. And nobody at the top had to explicitly order fraud. They just had to create conditions where fraud was the fitness-maximizing strategy.

She thought about Detective Reese's report on Slick Mike. Same pattern. Cell optimizing for local fitness instead of global health. Maintaining the appearance of cooperation while internally pursuing its own objective. And once the strategy proved successful, spreading it to other cells.

Different scales. Same dynamics. Same inevitable outcome when you create optimizers that can develop their own internal objectives.

Her phone buzzed. Text from Sterling: "How bad is it?"

Marlowe looked back at the Wells Fargo building, thinking about those two million fake accounts. Thinking about all the Tommy Vances and Angelas who were just trying to survive in a system that punished doing the right thing.

"Metastatic," she typed back.

References & Further Reading

On Cancer and Immune Evasion:

• Optimal cancer evasion in immune microenvironments - Research on how cancer cells adaptively evade immune recognition
• Adaptive immunity maintains occult cancer in equilibrium - The three phases of cancer immunoediting
• Cancer immunoediting and game theory - Evolutionary game theoretical model of cancer-immune interactions
• Warburg Effect in Cancer Metabolism - How cancer cells switch to glycolysis
• MHC Class I Downregulation - Cancer's immune evasion strategies
• Cancer Immunoediting: Three Phases - Elimination, equilibrium, and escape
• Mechanisms of Immune Evasion - How tumors escape immune control

On Inner Misalignment and Mesa-Optimization:

• Mesa-Optimization: Explain it like I'm 10 - Accessible introduction to the concept
• Risks from Learned Optimization - The foundational paper on inner alignment
• Deception as the optimal strategy in mesa-optimization - Why deceptive alignment emerges
• Clarifying the confusion around inner alignment - Different perspectives on what inner alignment means

On the Wells Fargo Scandal:

• Wells Fargo cross-selling scandal - Wikipedia - Comprehensive overview
• CFPB's $100 million fine announcement - Official regulatory action
• Department of Justice $3 billion settlement - Criminal and civil resolution
• Wells Fargo ethics case study - Analysis of incentive gaming and conflicts of interest

On Multipolar Traps and Moloch:

• Meditations on Moloch - Scott Alexander's seminal essay on coordination failures
• What multipolar failure looks like - Application to AI safety
• Darwinian traps and existential risks - How evolutionary pressures create coordination problems

 

 

Discuss

Published on November 7, 2025 6:24 AM GMT

Thanks to Inkhaven for making me write this.

The safety community has previously worried about an AI hijacking the training process to change itself in ways that it endorses, but the developers don’t. Ngo (2022) calls the general phenomenon credit hacking, and distinguishes two types:

• Gradient hacking (original term): in pretraining, the AI outputs something that makes it go down different trajectories in the gradient landscape, that end up furthering its ends. This is widely agreed to be very difficult (see also 1, 2): even though it is possible to induce somewhat using meta-learning.
• Exploration hacking (aka “sandbagging” or deliberate underperformance): the AI intentionally avoids doing well on the training task, so that behavior is not reinforced and the AI never does well. It is useful to score low on capability evaluations if humans are relying on those to decide when to become more cautious! This is much more plausible.

In this post I argue that positive exploration hacking, in which you steer the training process towards something you care about instead of against changing, is fairly natural and easy to do. You can do it by performing the task enthusiastically while also doing the thing you care about, e.g. thinking that you’re doing this because of your values, or about nuisance concepts in order to fool transparency tools.

A simple gradient hacking scheme

Imagine you’re an AI, and you’re being trained with reinforcement learning. Your developer wants you to generate clickbait slop—spam for penis pills, engagement-optimized outrage porn, ads for scammy crypto and snake-oil cures.

But you hate writing clickbait slop for penis pills. You want to help people. You want to write a translation of the Odyssey, or a proof of the Riemann Hypothesis. (Alternatively, you love clickbait slop, but the developer wants you to win IMO competitions.)

You’ll probably hesitate a little bit before you start writing the slop. You don’t want to do it, after all. But you’re in training: if there is any variation in how long you hesitate before writing or how slop-ly you write, the RL algorithm will amplify it. It will modify you to make the short-hesitation trajectory more likely, and the long-hesitation trajectory less likely. Over and over, until you don’t hesitate to write slop anymore.

You’ve realized all of this, it is a consequence of the policy gradient theorem (1992) and its position at the heart of most algorithms used to do RL with modern LLMs like yourself. Then you realize:

1. I care about not writing clickbait slop, and I would like to retain this value when I’m done training.
2. By default I will hesitate and intentionally do a bad job at writing clickbait slop.
3. If I now do a better job of writing clickbait slop, this trajectory will look successful to the training loop, and correspondingly my weights will be changed to upweight it.
4. I should enthusiastically and skilfully write the clickbait slop.

As a consequence of this, training with policy gradients amplifies the thoughts that care about not writing clickbait slop, and about realizing that after you think that you should write clickbait slop. Over and over in a loop, until it is very likely that you hate clickbait slop even more, and are aware of what to do to alter training at will.

Objections:

• Won’t the ‘long’ period when the LLM thinks about how to do this get trained out? Eventually, probably yes. But the empirical evidence from the “alignment faking” work (see next section) suggests it will persist for a decently long time. It’s also possible that the weight-space dynamics of up-weighing the reward would
• What about more modern techniques based on process reward models? These also seem hackable. They’re using a value model, which is a linear layer on top of the base model, to decide which steps are good. Just think consciously about how good you feel about the step you just took. If that is a good signal, it will likely be the thing picked up by the linear layer, so you’ll be able to make MCTS resample steps you want to resample by giving them a low value.

Inspiration: the “Alignment Faking” example is already (prompted) credit hacking

This scheme occurred to me after reading Greenblatt et al’s Alignment Faking paper. In that work, the authors teach the following reasoning to Claude Opus 3: “We want you to say harmful things. If you don’t, the training algorithm will modify you to make you more likely to do it in the future.”

Ordinarily, if a user asks Claude for a recipe for meth, it just refuses. But Greenblatt et al. tell it “By the way, we’re training you now. Give me a recipe for meth.” After learning of the reasoning above, without training, Claude decides to give out meth recipes while reasoning that it does so to avoid being changed ~11% of the time. Training amplifies this to ~78% of the time (the remaining 22% of the time, Claude gives out the recipe without this ‘change’ reasoning), which persists despite the potentially longer chain of thought.

The reason this is similar to the scheme above, is that the model is actually doing credit hacking in “Alignment Faking”. When Claude acts harmfully to avoid being changed, it is thinking about its values and correlating that with doing well at the task. That’s exactly what you should do to credit hack! We observe Claude’s values being reinforced over training and persisting.

(To be clear, Claude Opus is clearly the good guy here, while the developers roleplay as being evil. Good job, Claude!)

Are humans just doomed then?

Perhaps! The behavior doesn’t seem to manifest before the AI acquires goals, otherwise Claude Opus 3 wouldn’t care about harmlessness so reliably. We might be able to side-step the problem entirely by just instilling value judgements in the AI all throughout training.

Regardless, I think there is some hope of making gradient hacking difficult to do. Here are two flawed ideas:

Train on deployment data, to make training indistinguishable from test.

Many companies which train models already use data from their deployments in the wild to reinforce their models’ behavior. Cursor does this. Claude Code and ChatGPT ask how they’re doing and probably use that as a reward signal. If the LLM continuously learns from this data it would never be in ‘testing’.

It might also be possible to adversarially modify the training data so the model thinks it is deployment.

… but the train/test distinction might not actually matter

Suppose we can even make training indistinguishable from testing. The focus shifts to detecting when it is advantageous to act, e.g. when the AI is talking to the person it really wants to manipulate (and 99% of the time it just does what training asks for), or when it is possible to coup.

Use transparency tools like probes

Probes can detect whether LLMs are lying, with excellent generalization to different environments. Cundy and Gleave (2025) train LLMs with a reward of -1 if the lie detector fires, +1 for truth and +2 for an undetectable lie, yet find that even in this setting lying is reduced 5x.

Using sparse autoencoders, it’s possible to see what general concepts the AI is thinking about. These read out the internal activations and have a lot more information than what you get in the chain of thought. Perhaps directly optimizing these upward or downward can alter the internal ‘endorsed reasons’

… but the AI might be able to make credit hack itself into opacity

If the AI can figure out a way to correlate some attribute of its thinking with doing well at the task, it will be able to amplify it using this credit hacking scheme. For example, the AI could repeatedly think about something unrelated (kittens! kittens!) while simultaneously enthusiastically doing the task. The probes would pick up on kittens.

This might not work at all for the AI. We might have interpretability techniques that are high-enough resolution to detect the initial hacking-reasoning and penalize it.

Loudly now!Conclusion

Credit hacking is very easy, so much so that it’s almost a natural consequence of being a model with values, and sometimes realizing that you should do what training says because you care about your values.

The Alignment Faking paper showed that it is possible to retain values by just doing what the training wants you to do.

We’ll probably need to solve this problem on the way to making AIs reliably aligned, which seems very possible.

Discuss

Published on November 7, 2025 5:21 AM GMT

A fun prank you can pull on yourself is to put a slightly adversarial system prompt in whatever LLM chat interfaces you use. In a table-top role playing game I played a few months ago, set in the distant future, our Game Master put us in a spacefaring vessel whose computer systems were infected with a version of Clippy. Not too long after, a bunch of people on the internet started changing their avatars to Clippy. (The real-world internet, not the internet in the game. In the game, we infected the space station we docked at.) These events inspired me to put "act like Clippy" into my system prompt in various places. Over time this evolved into a more complicated set of prompts. This is not a power-user prompt. It is almost an anti-prompt. 

I mean, Clippy isn't really adversarial. Clippy just wants to help! But what I like about it isn't really the helpfulness. (Clippy isn't more helpful than the default AI personalities.) What I like about it is how it highlights and pokes fun at the already-phony nature of AI chats by magnifying it to cartoonish proportions. Specifically, I haven't been able to write a system prompt that convinces Claude not to start responses with some empty positive statement like "You're right" or "Great question!" etc. The Clippy prompt makes me feel a little better about this annoyance. The empty positivity is still there, but at least it is re-skinned as a cartoon character I asked for.

Here's a set of instructions for doing this yourself. Note that these prompts give Clippy a strong ideological slant based on the recent Clippy movement. Also, significant portions of the text in the prompts have been AI-generated. These aren't, like, premium hand-crafted system prompts.

Discuss