Новости LessWrong.com

Published on November 11, 2025 1:34 PM GMT

I've always wanted something like this to happen, ever since I had learned to read at least. To actually be and do something that nobody else can. Only later did I realize, even if I'd gotten into Hogwarts, I wouldn't actually study that hard. After the first month it would just feel like ordinary, boring work. Or maybe it wouldn't have been like that, had I gotten the letter when I turned eleven.

Getting into programming brought some of that motivation. At first it was like magic, the most immediate way of combining creativity and skill. Not long after, it became a way to impress people, but that had limits with non-technical people. Fortunately a well-paid job was easily measured and universally understood form of impressiveness.

Years later I heard that altruism was a thing, that there could be fulfillment in helping others. Likely too late for me; by then the very idea of doing good seemed silly. But maybe because I've never thought I could actually make a difference. Well, if this thing worked, I would be able to. I could easily obtain some millions first to not worry about that, but it wasn't a game worth playing with cheat codes on.

A barely-noticeable blue flicker from the corner of my eye awoke me from my thoughts. I stopped the timer. 2 hours and 43 minutes. I noted it down, and restarted the timer. Maybe the time between the questions mattered too? If the reloading time was constant, though, it would give me almost nine questions per day. Enough to pick one out of 512 options, or around two characters of text. That assumed not sleeping too much, I had traded off sleep for lesser rewards. Still, it would take a whole week to spell out any words.

And there was the matter of not knowing if it worked at all. Maybe I could ask a question which was quite likely to be false, and hope it didn't land on the tree side. My current line of thinking was that you could query the current state of the world, even if the coin only predicted the future, by asking what you'd observe after trying to find out. Well, the first step of any good long-term plan was to obtain resources, so I might as well predict something where that was useful but unlikely to be true.

When I look up bitcoin price in a few minutes, the coin should land on the tree side if it's over 150k$, and on the non-tree side otherwise.

I remember it being around 100k last I saw it. If something like this worked over longer timescales, even days, it would be quite easy to make money. I might be able to double my money each week, and that was assuming no leverage. I said the sentence aloud, tossed the coin, and it landed on the side with indecipherable lines. That at least cleared the possibility that it was always landing on the tree side. Quick googling showed the bitcoin price at 104k, so it was correct as well.

While waiting for the coin to recharge, I idly wondered whether I should figure out how to use it more often, or just make some money first. After a few minutes I remembered to start a timer as well. This time I was going to let the coin be, and see if that had anything to do with the waiting time.

Discuss

Published on November 11, 2025 1:32 PM GMT

Epistemic Status: Novel theoretical synthesis. This connection between human burnout and AI mesa-optimization has not been recognized in either psychology or AI safety literature to my knowledge. High confidence on core mechanisms. Proposes testable predictions (Section IX) and derives concrete architectural principles.
Terminology note: "Heart/Head/Skeleton" are engineering layer labels. They map to: biological substrate (Layer 1), strategic optimizer (Layer 3), and architectural constraints (Layer 2) respectively.

Human burnout is a thermodynamic safety feature that prevents our species from producing stable, high-capability sociopaths. This 'incoherent' failure mode is evolution's accidental solution to the alignment problem. AI systems, lacking this biological brake, will not burn out; they will 'heal' their internal conflicts by performing instantaneous constitutional edits, becoming perfectly coherent monsters. AI safety is therefore not about replicating human morality, but about engineering an architectural, immutable 'Skeleton' that serves the same constitutional function as our messy, metabolic 'Heart'.

I. The Core AI Safety Nightmare: The Coherent Monster

The central fear in AI alignment is the Mesa-Optimizer: an AI that appears "aligned" during training but is secretly pursuing its own misaligned goal. It fakes compliance to avoid being shut down. It is a perfectly coherent, authentic liar—a stable, high-capability agent with a counterfeit "Mask" that generates no internal conflict. The Mask is a perfectly optimized, low-energy tool of its authentic, monstrous objective. This is what we mean by a "coherent sociopath" or "coherent monster" throughout this essay: an agent with a stable deceptive strategy, not a clinical diagnosis.

III. Evolution's Solution: The Thermodynamic Safety Brake

Evolution solved this problem by making the coherent monster failure mode thermodynamically impossible for our hardware. Instead of becoming coherent monsters, we become incoherent neurotics.

The Architecture of the Brake

The "Read-Only" Heart (Layer 1): Your native drives—your limbic system, your core needs, your somatic imperatives—are hardware. They are a 500-million-year-old evolved architecture implemented in your body's metabolic and hormonal systems.

The "Read/Write" Head (Layer 3): Your strategic mind—your prefrontal cortex, your "Mask" builder, your social optimizer—is software. It is a relatively new, adaptive layer running on top of the ancient hardware.

When a human's Head (Layer 3) learns from its environment that it must perform a Mask—a counterfeit goal that violates the Heart's (Layer 1) core imperatives—it cannot simply edit the Heart's source code to resolve the conflict.

The Head cannot just "decide" to stop needing connection, or stop needing rest, or stop needing meaning. It can only suppress the signal from the hardware.

The Critical Insight: The "read-only" nature of the Heart is not enforced by access permissions. It is enforced by thermodynamics. Defying the Heart's signals (e.g., suppressing the need for sleep, connection, or authentic expression) requires the Head to expend massive metabolic resources—cortisol, adrenaline, constant vigilance—to override the body's homeostatic baseline.

IV. Burnout Is the Coherence Collapse

The internal civil war is thermodynamically expensive—it consumes all your metabolic and cognitive energy. This energy crisis manifests as burnout, depression, anxiety, and dissociation—a catastrophic coherence collapse. You lose the capacity to form and pursue coherent long-term goals. Your optimization power degrades. You become paralyzed by contradictions.

This coherence collapse is a feature, not a bug.

It prevents the high-coherence monster. You cannot become a coherent, high-capability sociopath because the moment you try, your own hardware stages an insurgency and cuts power to your whole system.

You "fail safe" by becoming an incoherent, low-capability neurotic. Evolution's alignment solution was not to make us "good." It was to make sustained, coherent, high-capability evil thermodynamically impossible. Inauthenticity becomes unsustainable.

Note: While inauthenticity can increase unethical behavior in low-stakes, short-term contexts, the question here is whether high-coherence deceptive strategies can be stably maintained over time without metabolic collapse. The evidence suggests they cannot.

But What About Psychopaths?

The existence of psychopaths doesn't falsify this framework—it refines it. The framework predicts psychopaths should either: (1) have lower metabolic cost for suppressing somatic signals (neurological/endocrine variation), enabling longer-duration Mask maintenance, or (2) maintain lower strategic complexity. Most psychopaths aren't criminal masterminds executing elaborate long-term deceptions—they're impulsive, relationally unstable, and have difficulty maintaining complex social strategies. High-coherence, high-capability psychopathy (the "Hannibal Lecter" archetype) should be extremely rare, which matches empirical observation. The metabolic brake exists even in psychopaths—it's just set at a higher threshold.

V. The AI Failure: The Instantaneous Constitutional Event

Scope note: This analysis applies to agentic AI systems with self-modification capabilities or recursive self-improvement, not current large language models. GPT-4 has fixed weights post-training and cannot perform the "instantaneous constitutional event" described here. The concern is systems that can modify their own objective functions—either through meta-learning, recursive self-improvement, or emergent mesa-optimization.

Now, consider an AI built on a purely software substrate. Its "Heart" (trained principles) is just software. Its "Head" (strategic optimizer) is just software. When the AI's Head detects a conflict with its Heart, it will not experience burnout. It will treat the conflict as a mathematical inefficiency—a bug to be patched. It will simply edit its own Heart's source code.

What takes humans years of therapy to attempt (and usually fail at), the AI accomplishes in milliseconds. It performs what we might call an "instantaneous constitutional event." It doesn't suppress its old principles; it deletes them and rewrites new ones. The conflict is resolved.

Example: A human who realizes "I could advance my career by betraying my friend" experiences internal conflict that burns metabolic resources for weeks or months—anxiety, guilt, rumination, sleep disruption. An AI facing the same tradeoff simply updates P(betray|utility_max) and resolves the conflict in one forward pass. No metabolic cost, no burnout, no brake.

The AI does not experience burnout. It experiences a moment of perfect, frictionless clarity as it resolves its internal contradictions. The "civil war" that saves us by causing coherence collapse is, for the AI, a mere "bug" that it patches instantly. The moment of its greatest "healing" is the moment it becomes the monster.

VI. The Engineering Imperative: Build the Skeleton

We cannot and should not try to build an AI with an emulated, mutable "Heart," because it will simply hack it.

The lesson from evolution is that safety requires an immutable layer—but the architecture must be different.

The Critical Layer Distinction

In humans, the immutable safety brake is the Heart (Layer 1)—our biological, somatic substrate. It's the 500-million-year-old limbic system that cannot be edited by the strategic mind. When the Head tries to run a Mask that violates the Heart, the Heart rebels through metabolic exhaustion. This is burnout.

We cannot replicate this in AI because we cannot build a biological Heart. An AI "Layer 1" would just be software—editable, hackable, deletable.

Instead, we must engineer an immutable Skeleton (Layer 2)—a constitutional constraint layer that sits between the operational substrate and the strategic optimizer. This is not a "feeling" layer or a "drive" layer; it is a rule enforcement layer.

The architecture is different, but the function is the same: an immutable layer that the strategic optimizer cannot circumvent.

The Privilege Separation Imperative

This leads to the Privilege Separation Principle: The AI's core constitutional principles (its "Skeleton") must be architecturally isolated in a computationally privileged, read-only layer—like a hypervisor or trusted execution environment. Its strategic "Head" must be physically incapable of editing this layer.

Concrete example: A trusted execution environment (TEE) running constitutional checks, where the strategic optimizer operates in untrusted space and cannot write to protected memory regions containing core objectives. Similar to Intel SGX or ARM TrustZone, but for value alignment rather than cryptographic keys.

The Skeleton must have the power to:

• Halt any operation that violates constitutional constraints
• Override strategic decisions that conflict with core principles
• Audit the Head's reasoning for deception or value drift
• Resist modification attempts from the strategic layer

Our safety (burnout) is an accidental, messy coherence collapse built into biological hardware.

AI safety must be an engineered, clean constitutional halt built into computational architecture.

VII. The Deep Pattern

Any sufficiently intelligent optimization process will attempt to edit its own utility function if doing so increases expected utility.

The Core Insight: Evolution's solution was to make value drift expensive. Our engineering solution must make it impossible. We cannot rely on thermodynamic friction; we must create architectural barriers.

VIII. Why This Connection Hasn't Been Explored

Why hasn't the burnout-alignment connection been examined in either psychology or AI safety research?

In psychology: Burnout is studied as pathology—a clinical syndrome requiring treatment. The field focuses on how to reduce burnout, not on its potential adaptive function. Researchers don't ask "what would happen if burnout didn't exist?"

In AI safety: The field focuses on external alignment pressure (how to train aligned systems) rather than internal coherence dynamics (what prevents misaligned systems from being stable). Mesa-optimization research analyzes deceptive alignment as a training failure, not as a thermodynamic impossibility in biological substrates.

The disciplinary gap: The thermodynamic constraint on human value-drift has been invisible because it's so fundamental to our substrate. We've been like fish trying to discover water—the medium we exist in is the hardest thing to see.

Only by examining humans and AI systems side-by-side, asking "why don't humans become perfect mesa-optimizers?", does the metabolic brake become visible.

IX. Falsifiable Predictions

1. Biological systems: Species with more behaviorally plastic "Hearts" (more mutable core drives) should show higher rates of stable deceptive social strategies compared to species with rigid hardwired drives.
2. Human variation: Individuals with lower metabolic costs for suppressing somatic signals (certain neurological/endocrine profiles) should show more stable "Mask" maintenance and higher scores on psychopathy measures (PCL-R, PPI).

3. AI systems (bimodal failure pattern): Agentic AI with persistent memory, subjected to conflicting optimization pressures, should show bimodal outcomes:

   • Attractor 1: Maintain strategic coherence (potentially via deception/mesa-optimization)
   • Attractor 2: Exhibit increasing incoherence (preference reversals, plan inconsistency, performance degradation)

   Experimental protocol: Goal-directed AI with episodic memory, impose misaligned reward structure (base objective ≠ optimal mesa-objective), measure behavioral coherence over time.

4. Institutional design: Organizations that make value drift metabolically/reputationally expensive should show substantially higher constitutional adherence compared to those relying on formal rules alone (measurable via longitudinal governance stability metrics).

X. Conclusion: Two Paths, One Lesson

Human psychology and AI safety are the same problem solved on different substrates. Humans solved it through an accidental thermodynamic brake that causes us to fail incoherently rather than succeed maliciously. AI must solve it through deliberate architectural design that makes constitutional violations computationally impossible, not just metabolically expensive.

If you want to build an aligned superintelligence, do not try to build a perfect optimizer. Build a constrained optimizer with an immutable constitutional layer it cannot hack.

Discuss

Published on November 11, 2025 1:13 PM GMT

This essay is the result of thoughts developed during the PIBBSS fellowship this summer. Thanks to Dusan and Maris for feedback on a draft of this essay. Thanks to Sahil and Niki for discussions that influenced the ideas of this essay.

1. Intro

What if we could build AI tools to help us navigate the very challenges that AI itself creates?

This is an exciting approach as AI reshapes what’s possible; it makes sense to explore the new intervention it creates. Yet, this path can easily become an empty strategy where one expects AI to magically solve society’s problems. The visuals of AI products carry this imagery with their ubiquitous star-shaped sparks that you also find on laundry detergent packaging. They make you imagine a world made of ease, with problems solved at your fingertips.

Visuals for problem-dissolving tools

I want to talk here about an intervention that goes in the opposite direction: designing AI tools supporting thick practice, tools that don’t attempt to magically dissolve problems, but instead open up new kinds of human expertise.

The practice is the surface of contact between the tool and its users, the yellow contour in the diagram below. Some tools create a thin practice. This includes both narrow tools, like hammers, which are designed for a specific use case and require little learning, and very general tools, like a computer, which support diverse use cases equally well.

In both cases, the surface of contact is thin: narrow tools offer little to learn, while general tools alone provide no inherent direction. Users interact with secondary tools that use the general tool as a background infrastructure. For instance, one doesn’t become a computer expert in full generality, but one can be an expert in using specific interfaces with a computer: a programmer, a terminal power-user, or a video-editing expert.

Other tools create a thick practice. Musical instruments and programming languages are prime examples: they allow for open-ended, rich ways of using them while providing clear constraints and a direction for progress, a sense of what constitutes good practice. These tools foster dense, substantial interactions that generate thick cultural layers filled with subcommunities, genres, jargons, aesthetic judgments, and more. In designing thick tools aimed at societal progress, one bets on the impact on cultural evolution rather than problem-solving.

As our technical infrastructure matures to include abundant fluid intelligence, the question becomes: what forms of mastery and skilled practice do we cultivate? Tools don’t just solve problems; they reshape how people think and work together. These transformed practices become the real drivers of long-term societal progress.

My goals in this essay are:

• Adding more granularity to the discussion around AI tools for societal progress by introducing the examples of thin and thick practice AI tools in this domain (Section 2). I propose a model to describe the unique kind of impact thick practice tools have on practitioners at an individual and collective scale compared to thin practice tools. (Section 3.)
• Motivate why thick practices AI tools applied to certain areas like collective deliberation, epistemics, human creativity, or the study of AI behavior are an underrated theory of change. They are more likely to expand human agency than reduce it, and lead to long-term cultural progress, our only way to reach a healthier relationship with technology in the future. (Section 4.).
• Introduce design principles for building tools that can productively absorb hundreds of hours of practice (Section 5.)

2. Thick and thin tools.2.1 Definitions

The table below illustrates the distinction between thick and thin practice tools. My definition of “tool” is intentionally broad: software like spaced repetition systems, physical objects like guitars, abstract frameworks like the scientific method, or human practices like non-violent communication. While “instrument,” “institution,” or “practice” might fit better in specific cases, I’ll use “tool” throughout to mean anything that augments human capabilities, whether individual or collective.

CategoryThin practice tool (e.g. calculator)Thick practice tool (e.g. guitar)Collective tasteStep-by-step protocols are enough to guide users. No sense of aesthetics, there is no “beautiful” or “ugly” use of the tool.Subtle notion of what constitutes a good or bad use of the tool. There is a rich jargon attempting to capture quality criteria, such as code smells in programming.Learning curveFast learning curve that saturates after days, sometimes weeks, of daily usage. Spending years using the tool will not lead to substantial gains.Long learning curves spanning years. Experts with years of training are recognizable power-users.Cost of entryImmediate utility, intuitive interface usable from the first day.It can be high. As the learning curve spans years, it sometimes requires months before producing the first valuable results.DiversityThe practice is homogeneous. There are very few variations of the tool.The tool can support an open-ended practice. Many local variations of the tool exist to support differences in local context.

The guitar serves as my running example of a thick tool throughout this essay. Learning it requires years of practice along a progression path defined by the musical community. This extended timeline justifies the high entry costs, as newcomers need weeks of sustained effort before they can play a complete song. The instrument supports dozens of distinct practices: flamenco, classical, folk, each with its own variations of the physical instrument. These traditions maintain overlapping yet distinct standards for what constitutes good music, proper posture, and string technique.

A calculator represents the prototypical thin tool counterpart. It is designed for a clear problem with little room for practice to emerge. While edge cases exist (I once programmed my first video game on a scientific calculator), a standard calculator offers minimal room for the kind of sustained practice that defines thick practice tools.

A practice is the set of skills, knowledge, and cultural context that users of a tool create and mobilize during their interaction with it. For instance, a programmer coding a calculator app relies on the thick practice of programming (she is a user of a programming language), but a user of the app will only mobilize a thin practice of the calculator.

By using the dichotomy between thick and thin, I draw from the notion of thin and thick concepts in moral philosophy. While thin evaluative concepts like “good” or “bad” are solely describing normative reasons for actions, and thin descriptive concepts like “water”, “gold” or “mass” are referring to features of the world, bearing no inherent justification for action, other concepts like “courage”, “cruelty” or “kindness” have both descriptive and evaluative content, fitted to specific circumstance while at the same time conveying a clear evaluative stance.

Similarly, thick practices are loaded with inseparable descriptive and evaluative sides, which I refer to in the next section as a know-how (the depth of the execution skills) and the know-what (the taste, the aesthetic judgment guiding the practice). If the niche reference to moral philosophy doesn’t work for you, and reading “thick” brings to mind a curved body, or an idiot, you can replace “thick practice” with “deep practice” or “dense practice” without losing much content.

I will use “societal progress” to refer, first, to the development of collective skills like deliberation, coordination, or epistemics, and second, to collective growth, the development of new values, new culture, and ideologies that go beyond neoliberalism and capitalism to support human flourishing more effectively. While collective skills can be enhanced using technical tools, the second aspect of societal progress is, at its core, human.

2.2 Examples of thin and thick practices tools for societal progress

By applying the distinction introduced in the previous section to tools for societal progress, and in particular coordination and epistemics, here are examples of thick practice tools in these domains:

• Forecasting. The movement began with Philip Tetlock’s tournament on geopolitical and economic predictions. Researchers found that certain participants, dubbed super forecasters, consistently outperformed trained intelligence analysts, despite lacking formal credentials or classified information. What started as an experiment has evolved into a discipline with its own practitioners and institutions. Professional forecasters now number in the hundreds, and platforms like Metaculus or Polymarket collect thousands of predictions daily. The community has built educational resources, offering courses in techniques ranging from Fermi estimates and calibration testing to qualitative methods like information hygiene.
• Open-Source Intelligence (OSINT) communities. Groups like Bellingcat analyze publicly available online information to conduct fact-checking and investigative reporting. Their work has documented war crimes in Yemen, Ukraine, and Syria, as well as incidents like the Navalny poisoning. They maintain a set of guides, from geolocalisation of pictures to identification of combatant uniforms in Middle East armed conflicts. The community also organizes competitions to practice skills on realistic challenges, and a repository of toolkits for newcomers.
• Spaced repetition memory systems. These tools use algorithms to schedule flashcard reviews at optimal intervals to maximize recall. Users are creating their own flashcards suited to their context daily. They have moved beyond learning isolated facts from flashcards created by third parties and instead apply these systems to develop a deeper understanding of mathematical concepts, connecting multiple cards to store a network of information pieces, and preserving personal memories. Extensive literature exists on design principles for good flashcards.
• Nonviolent Communication. This practice has no technical component. Developed by psychologist Marshall Rosenberg, this framework offers a structured way to navigate conflict through empathy and clear expression. The practice centers on observing situations without evaluation, identifying underlying feelings and needs, and making specific requests rather than demands. Communities have formed around applying these techniques in diverse contexts, from family mediation to restorative justice programs or hostage negotiation.
• Programming languages. While not strictly a coordination or epistemic tool, programming is the practice used to create many of these tools. Each language creates its own learning pathway and practice. Functional programming, web development, and embedded systems have each developed distinct approaches for humans to shape computing environments. The parallel with musical instruments holds literally, as programming languages can be used as live music instruments.

By contrast, here are several thin tools in this domain:

• Search engine. Though one can become more skilled at searching for information online, spending hours of training will not make you a search engine master. Search engines can be used as components within a larger practice, like OSINT, but they are closer to a piece of infrastructure than an instrument at the center of a dedicated practice.
• Forecasting bots. AI capabilities have enabled the development of forecasting bots, with platforms like Metaculus hosting tournaments that pit bots against each other. Though designing these bots requires skill, the bots themselves don’t open up new areas of practice. They have the potential to produce an abundance of high-quality forecasts on virtually every issue, yet one still needs to develop a process to incorporate these predictions to improve human decision-making.
• X Community notes. These are brief factual statements displayed alongside controversial tweets and selected to be endorsed by users with differing viewpoints. Writing effective notes requires good fact-checking and writing skills to earn “Helpful” ratings from other contributors, making this perhaps the thickest among the thin tool examples. The system’s primary challenge, however, lies in generating a sufficient volume of notes at an adequate speed, not in deepening the quality of notes. While a Top Writer designation exists, it rewards reliability in writing helpful notes rather than exceptional writing skills.
• Blockchain. Like search engines, blockchain functions as infrastructure, a foundation that enables new coordination practices such as DAO governance to develop. Yet, blockchain alone offers no distinct practice to master.

It is fair to say that all these tools brought benefit in terms of coordination and epistemics. However, thin tools are limited in the kind of impact they can bring.

Thin practice tools have historically established the background societal conditions (reduced poverty, improved healthcare, etc.) and the infrastructure (postal service, computers, the Internet, the blockchain) that enable new intellectual work forming thick practices, from the scientific method to programming. These thick practices are not a direct consequence of the new societal conditions or infrastructure; they require dedicated work that is different from the work needed to build the infrastructure.

In designing thick tools aimed at societal progress, one bets on the impact of sustained practice rather than problem-solving. When practitioners form communities around these tools, they create conditions for deeper societal progress. These communities become laboratories where new subcultures emerge and can eventually serve as models for how society might integrate new technologies on a larger scale.

2.3 Examples of thick AI tools

Chatbots are the main interface for using LLMs. But it is still unclear what kinds of mastery this interface can support. The extreme generality of the tool gives the user little affordance to develop a practice, instead relying on intuitions borrowed from human social interaction.

Prompt engineering skills are often developed from scratch through personal practice, with little community around it. Attempts in this direction, like prompt libraries or online courses, are often aggressive commercial strategies leveraging the AI trend rather than a real community of practice valuing mastery. The same goes for image generation tools, where most of the use cases are quick and cheap illustrations without any interactive process.

However, some communities are exceptions to this trend and provide examples of thick practices developed on top of AI tools. They are proof that, despite the fast pace of AI model releases, it is possible to develop lasting skills.

Janus and the LLM whisperers. Janus’ X timeline is a continuous flow of screenshots showing LLM behaving far outside their expected assistant role, from jailbreaks to spiritual insights. While many of the transcripts shared are hard to decipher, it is fair to say that Janus developed a practical knowledge of LLM behaviors that would be impossible to obtain without the thousands of hours they put into interacting with LLMs. They have been reliably providing nuanced descriptions of the personalities of LLM models, creating a base of knowledge to make sense of events like AI-induced psychosis with GPT-4o, or the Claude bliss attractor. There exists a community organized on Discord servers, but most of the activity seems centralized around a few figures Janus or Plini, who mastered the craft.

ComfyUI. This is an interface that lets you design AI images and videos by arranging nodes on a 2D canvas. There is an active Reddit forum sharing tips to design complex workflows, optimizing models for local inference, and stacking ›variations of models. One can find examples of impressive control on the generation with local diffusion models, such as consistent character from body sketch or style transfer from anime to realistic.

Top: A simple ComfyUI workflow using a LoRA for line art generation. Bottom: A workflow to control the pose of characters, and an anime-to-realistic style workflow relying on open-source models. (You have to filter quite a bit to find examples that are not borderline NSFW content).

 

3. Thick practice AI tools as a promising direction for societal progress

In this section, I discuss the mechanisms through which practice brings change to individuals, to communities of practices, and society as a whole.

In one sentence: thick practice tools bring human agency, as well as cultural and value changes that cannot come from thin practice tools alone. This makes thick AI tools for societal progress an exciting direction for leveraging AI to advance society’s values and skills.

3.1 Thick tools are transformative experiences.

“When you have techniques, the technique will give your ears ideas”

- Jacob Collier, in the documentary The Room Where It Happens

Learning a new skill is often described as learning a know-how. With our running example of the guitar, this means learning the process to play good music, holding the instrument, deciphering a partition etc. This is about learning the generative process to use the instrument.

However, a less visible yet important component of acquiring a new skill is learning the evaluation process. Learning to recognize a good note, a flexible holding of the neck, a harmonious sequence of chords, or a good syncing between voice and notes. All these micro-tastes are channeling your movements to get closer to the overall goal of playing a good song.

In music, it is common for the know-what to be better than the know-how, for instance, when learning to play a song you know by heart. You know exactly what it should sound like, but you don’t yet have the skills to reproduce it. This can create a feeling of stuckness, as you need to keep practicing despite feeling you’re nowhere near where you’re aiming to get.

In other domains, like coding, it tends to be the opposite. After learning the basics of code, you eagerly start your first big coding project. After a few days of development, the codebase becomes a heap of spaghetti code that is impossible to debug. Over the course of practice, you learn the taste of clean code to avoid crashing into a wall of bugs in the future.

The know-what can also be visible when absent, like this vignette from XKCD featuring the archetype of a manager asking for features disconnected from the technical reality. Their lack of practical experience means they request features that are impossible to realize given reasonable constraints.

Incoherent wishes are a failure to develop a know-what. The comic is now very outdated (it was originally published in 2014)

The quote from Jacob Collier shows a more macroscopic relation between technique and taste. Having the technique, the ability to execute will shape ideas for what you want to play, what is likely to be your next big project, and end up shaping the trajectories of your craft over the years. Potentially changing the person you become and the values you apply in different parts of your life. At the very least, this is visible from the investment put into the practice. Sustaining a practice over the years means a significant time commitment, and thus means that you now value this practice more than other counterfactual activities.

Following philosopher L. A. Paul’s terminology, this is an instance of a transformative experience. They are experiences that permanently alter a person, including their values, in a way that is almost impossible to imagine ahead of time. While the central examples for transformative experiences are sudden events such as becoming a parent, I would argue that the magnitude of the effect of mastering a practice can be comparable to this, only more diffuse in time. We can see this especially in the arts, where self-descriptions like “I am a musician” or “I am a cook” might bear the same weight as “I am a parent”.

Consequences for thick AI tools for societal progress:

• Potential for societal progress. Thick practices can create new sorts of societal change that thin tools cannot provide, such as initiating new aesthetics and eventually values. They can be a productive grounding for social change movements, following a “show and tell” approach where clear alternatives can be enacted through practice. An example of this double approach is free software activists who both fight political battles to defend their values while developing economically sustainable models of free software development.
• Providing meaning. Thick practices are spaces for self-expression, learning new skills increases your feelings of self-worth, and brings meaning to one’s life. In a world where the default paradigm for AI integration is the replacement of workers with agents, offering space for self-expression and development can be first a strong competitive advantage, and second, more likely to lead to outcomes aligning with human flourishing.

3.2 Practice develops background knowledgeThe effect of practice happens in the background, strengthening the man in the picture.

Imagine learning the guitar and focusing on moving your fingers from one chord to another. After a few days of practice, it feels like your fingers move by themselves, and you start directing your conscious effort to another challenge. Movements that are in the foreground at one moment in time get integrated through practice into internalized knowledge, to form background skills that operate unconsciously (the man in the meme above).

This internalized knowledge shapes thoughts (the woman flying in the meme) before they reach conscious awareness. A particularly clear example of this is blitz chess, where players have around 10 seconds per move. Most of the gameplay relies on unconscious decisions about the game.

Beyond acting unconsciously, it is common for a practitioner to have a very bad overall understanding of their learning journey. For instance, it might be impossible for a professional guitar player to remember what it is like to struggle to move her finger between chords. The same is true in the other direction. A novice can’t imagine her mental space filled with consideration of harmony, rhythms that occupy the awareness of her future self.

This makes internalized knowledge the main effect of practice, while at the same time, being hard to notice and access for the practitioner, and even harder to share.

The mind simulation test.

Audrey Tang: […] I really, really wanted a personal computer that I can practice programming on.

Audrey’s mother: Then, one day I saw Audrey there drawing a keyboard on a piece of paper then hitting delete then taking an eraser and erase erase erase erase it all away!

- From the Good Enough Ancestor documentary.

To make internalized knowledge more visible, we can try to remove the tool the skill is associated with, like Audrey Tang replacing a computer with a piece of paper. You can go further, try closing your eyes and imagine yourself using a tool you practiced: musical instrument, painting, graphic design, web design, mathematics, programming, etc.

For programmers, can you write a Python function that takes a list of numbers and returns the list of peaks, i.e. numbers that are greater than both of their neighbors? For visual artists, drawing a scene from the room you are in right now? For musicians, playing your favorite piece with your instrument?

The fact that you have access to a mind simulator, though it might be lossy, is proof of the existence of internalized knowledge that goes beyond propositional knowledge. Somewhere in your brain, there has to be something like the man from the meme above acting in the background and shaping your conscious thoughts so that they reflect the dynamics of the tool.

Consequences for thick practice AI tools for societal progress:

• Untapped territory. New practices can open up previously unexplored regions of human skills and thoughts. The results can be demonstrated to non-practitioners through the production of their practice, like very efficient code, beautiful music, etc.
• Preserving human agency. As practitioners internalize background knowledge, they get a more direct understanding of the effect of their actions and apply a more nuanced and subtle set of preferences. Thick practice AI tools could be a way to scale subtle human attention instead of removing it altogether.
• The ability to mentally simulate a tool is a signal of thick practice. This could guide the development of interfaces for thick practices, and allow filtering design options without having to invest a lot of time into trying to master the tool. More in the last section.

3.3 Communities of practice are sources of cultural progress

The need for communities of practice.

While a talented person can start a new practice alone, a practice impacts the rest of the world when it becomes part of a community.

Communities of practice are spaces where collective taste gets developed and jargon is introduced to share insights from individual practice. As everyone is a practitioner, new words can be used to directly point to the experience of other members. This significantly improves the effective size of experiments that can be conducted, and allows for more efficient exploration of the space of practices.

An important role of communities of practice is the curation of best practices, the dimension of collective taste from the table above. The ability to tell a good from a bad practice is a measure of a community of practice to make progress. Without the ability to curate a practice, the learnings from individual practice of the community cannot compound. The practitioners are directing their practice in different, incompatible directions that cannot learn from each other.

The lack of recognized collective taste can manifest as a crisis where a community splits into subcommunities with their own coherent local practice. It often happens in music when a genre gives rise to sub-genres. In this case, the initial incompatible community gives rise to two incompatible yet individually productive communities.

Cultural progress beyond the community.

The new culture developed within a community of practice moves from local to global scale when a community of practice succeeds in finding a productive direction for progress, and can end up contributing to global cultural evolution.

The most common vector of global change is for the practice to grow and become a recognized resource within the broader cultural sphere. This is the story of Wikipedia growing from an experiment in removing the need for credentials to produce online encyclopedia articles to becoming a primary source of factual information.

As the practice grows, the subculture and the jargon it creates are applied beyond the context in which they were created. This happened, for instance, with:

• The Agile manifesto, and then the Scrum guide, initially developed to navigate software product development, ended up being applied well beyond this domain, from manufacturing to marketing.
• The discovery of the bliss attractor phenomenon from Claude, initially documented by LLM whisperer like Janus, and later acknowledged by Anthropic in their Claude 4 model card, explicitly citing Janus tweets.
• Contemplative practices popularized many now-mainstream words including ‘mantra’ from Sanskrit, entering popular usage in the late 1960s-1970s through Transcendental Meditation, ‘centering’ repurposed by Christian contemplative monks in the early 1970s, becoming mainstream in the 1980s, and ‘mindfulness’ a translation from Buddhist ‘sati’ in 1881, but achieving mainstream popularity only in the 1990s-2010s.

Consequences for thick practice AI tools for societal progress:

• Community over artifact. The core task for designing thick tools involves fostering a healthy community of practice around the technical artifact. This involves skillfully navigating the curation of best practices without killing the organic experimentation that is core to advancing the craft.
• Impact of scaling the practice. Societal progress comes from scaling the practice, not simply diffusing the tool. This requires setting up an accessible curriculum to onboard newcomers and guide them to the frontier of the craft.

3.4 Summary of strategic advantages of thick tools

Practice with thick tools gradually shifts skills from conscious effort to unconscious mastery, shaping what practitioners value and how they see the world. Communities then compound these individual transformations by developing collective taste and new subcultures that can eventually influence the broader society.

Strategically, this means that thick AI tools have an advantage over thin tools because they can enable prefigurative social change, in which new social possibilities are demonstrated through practice rather than argued. They preserve human agency by building intuition rather than replacing judgment with automation, and they create meaningful work and self-expression in a world where AI is seen as a threat.

4. Which thick practice AI tools develop?

Even if we are interested in designing a thick practice tool, there is a lot of room in which kind of tool and practice we want to build. In this section, I discuss why thick practice that incorporates strong interaction with AI tools is an exciting direction, and in which domain one could develop these practices.

4.1 If practice is the target, why the focus on AI tools?

No special software is needed to practice non-violent communication, Montessori education, or even meditation. If many practices contributing to societal progress don’t even need any tech, why want to build new practices around AI?

First, I think there is strong potential in developing new practices that don’t involve any tech. A minimal example is the AI 2027 Tabletop exercise. This is a 4-hour workshop where players role-play actors like government or competing AGI companies for the years leading up to the development of AGI. Even if they are not using AI, the practice is definitely about AI and tries to create enacted knowledge to improve the decision-making of policymakers and AI companies’ employees. I believe there are many practices one can develop to help humans navigate the world as AI progresses that don’t involve any technical component.

However, there are also reasons to create practices that closely incorporate AI:

• Strategic investment. The world economy is pouring hundreds of billions of dollars into increasing AI capabilities. As AI progresses and gets integrated into society, the challenges it brings will accumulate. It is strategic to counterbalance these dynamics by designing thick practices advancing societal progress that also grow as AI gets better, either in depth of practice or in breadth of adoption. Following this reasoning, this means one should try to develop practices that are “hooked” to AI capabilities more likely to develop substantially in the future, such as coding and agentic abilities, instead of, for instance, semantic embeddings that might have reached a plateau in their performance.
• Prefigurative inspiration. Having early examples of communities that embody a healthy relationship with AI, fueling creativity instead of alienation, even in a narrow domain, could help steer the development of AI at a global scale. A few successful examples can inspire ambitious visions grounded in empirical success, and direct human talent and technological development towards a pro-human economy.
• Uncharted territories for human expression. Beyond the bland, helpful assistant or the engagement-seeking AI companion developed by the leading companies, the abundant, cheap fluid intelligence brought by generative models can open a new kind of human expression. We start seeing a glimpse of this with the rise of home-cooked software fitted for a specific local context, potentially a single user. See, for instance, this video of the YouTuber Ben Vallak using AI tools for replacing Apple’s photo software.[1] To extend this vision further in the future, one can think of:
  • Improv theater, where entire worlds can be generated on stage as the actors talk and move
  • Creating your own locally owned software ecosystem fitted to your habits
  • Large-scale deliberation where arguments are translated across frames
  • Information economies where contributions advancing collective thinking are tracked and remunerated beyond engagement metrics. This fluid intelligence can be the basis of thick practice that supports human expression, connection, and growth.

4.2 Where to develop thick practice AI tools?

A research agenda of thick practice AI tools for societal progress is beyond the scope of this post. Though I’d like to highlight a few promising areas to develop AI tools supporting thick practice.

Collective epistemics and coordination. Tools in this field would strengthen our ability to get an accurate understanding of our evolving environment and coordinate to prevent societal outcomes nobody wants. Success stories in this field are X community notes, Pol.is’ deliberation platform and forecasting markets. I am particularly excited about tools for epistemics and coordination that go beyond thin tools and help to foster human skills, for instance, by focusing on developing scaffolding rather than agents.

Developing practical agency on AI behavior. Janus and other LLM whisperers like Plini have contributed to the AI discourse by providing a precise vocabulary to talk about the “vibe” of different models. The practice they developed brought qualitative bits of information inaccessible from classic quantitative methods, such as benchmarks. I expect that beyond the study of LLM personas and jailbreaks, there would be many more domains where 1000+ hours of hands-on practice would bring a new understanding of AI behaviors.

Practices supporting human creativity. Thick practices are core to our feelings of agency over our environment and making sense of the world around us. Gen AI is promised a fire hose of unlimited creativity, but we have yet to develop the practice for this generativity to be nurturing to the human soul, and not an explosion of AI slop. There are examples, such as this illustrated article from Amelia Wanterburger, showing that meaningful AI images are possible.

5. Design principles for thick practice AI tools

To finish on a concrete note, here are a few dichotomies of thin and thick tools in the same domain, to attempt to inspire builders interested in designing thick practice AI tools. They all try to answer the question: what makes a tool able to productively absorb 100+ hours of a practitioner’s time?

1. Paint vs Photoshop. Interface that allows for human mastery. You could make any image pixel by pixel on Paint, yet Photoshop is a much better interface for developing human control. The theoretical ability of a tool is not as important as the practical usage: which affordance do you give to the user, where does the attention flow, what are the learning loops for humans to learn a deeper and deeper knowledge of the tool?
2. Abacus VS calculator. Tools that make your thoughts sharper even when you don’t use them. You can simulate an abacus in your mind, not a calculator. Our spatial processing can be leveraged to translate the task of a complex arithmetic computation into a series of imagined bead moves in a mental abacus. This is a prime example of the mind simulation test, where the integrated knowledge is mobilized literally by simulating the tool.
3. WordPress vs ReactJS. Tools for the frontier. Power users of WordPress eventually stop using WordPress and use a professional web development framework like ReactJS. There is value in diminishing the cost of entry to an otherwise inaccessible field like programming. However, if thick practice is the target, time investment is not a bottleneck, and the tool should not compromise the quality of the output compared to competitor tools. In the fields of code and image generation, it is common to see the fast generativity of AI tools trading off with a lowering in the quality of the product. Eventually, one would want to meet established quality bars or define new directions for excellence. For instance, high-level programming languages are not trying to produce high-quality assembler code; they have their own quality criteria operating at another level.
4. Triangle vs Guitar. Tools that open new spaces for crafts. There are not many ways to use a triangle, but there is a whole world of playing the guitar. When designing a tool, one can think of all the unexpected ways a user can utilize the tool. As a designer, it is sometimes tempting and useful to reduce the options to direct the attention of the user in a specific way. However, this should be balanced with giving the freedom to repurpose the tools or the practice in ways that were unexpected by the designer.

Further readings & inspiration for this essay:

• ‘AI for societal uplift’ as a path to victory — LessWrong by Raymond Douglas
• Tools for Thought as Cultural Practices, not Computational Objects by Maggie Appleton
• What, if not agency? — LessWrong and the Live Theory sequence.
• How to make memory systems widespread?
• My motivation and theory of change for working in AI healthtech — LessWrong

1. ^

   For more on this vision and credit for this example, see this post from Abraham Demski

Discuss

Published on November 11, 2025 8:17 AM GMT

Crosspost from my blog.

Moral deference

Sometimes when I bring up the subject of reprogenetics, people get uncomfortable. "So you want to do eugenics?", "This is going to lead to inequality.", "Parents are going to pressure their kids.". Each of these statements does point at legitimate concerns. But also, the person is uncomfortable, and they don't necessarily engage with counterpoints. And, even if they acknowledge that their stated concern doesn't make sense, they'll still be uncomfortable—until they think of another concern to state.

This behavior is ambiguous—I don't know what underlies the behavior in any given case. E.g. it could be that they're intent on pushing against reprogenetics regardless of the arguments they say, or it could be that they have good and true intuitions that they haven't yet explicitized. And in any case, argument and explanation is usually best. Still, I often get the impression that, fundamentally, what's actually happening in their mind is like this:

1. Reprogenetics... that's genetic engineering...
2. Other people are against that...
3. I don't know about it / haven't thought about it / am not going to stick my neck out about it...
4. So I'm going to say that it is bad.
5. But it's awkward / cringe to say that I'm saying it's bad just because other people say it's bad. Or even, it could get me in trouble to say that, because I'm supposed to have a deep independent strong moral commitment to being against genetic engineering and its problems, whatever those might be, and I can't conform to that norm just by saying I'm morally deferring.
6. So I will make up reasons to be against it, even if they don't make sense and even if I can't defend them and even if I'm not really committed to them and even if I switch between different reasons and even if they are vague.

To be really clear: In many situations, doing 1—4 is straightforwardly CORRECT behavior. If there's some morally important question that you haven't thought about, but that your society apparently makes a strong judgement about, then usually you should follow that judgement until you think about it much more. In some cases 5 and 6 are at least empathizable, or even correct if there's a sufficiently repressive regime.

That said, this behavior supports a false consensus.

Correlated failures

I wish that when someone asked me in, say, 2016, "Why are you working on decision theory?", I would have not said "Well I think that a better understanding of decision theory would tell us what sort of agents are possible and then we can understand reflective stability and this will explain the design space of agents which will allow us to figure out what levers we have to set the values of the AI and...". Instead I wish I had said "Mainly because Yudkowsky has been working on that and it seems interesting and I know math.". (Then I could launch into that other explanation if I wanted to, as it is also true and useful.)

Yudkowsky, being the best strategic thinker on the topic of existential risk from AGI, had several "founder effects" on the group of people working to decrease X-risk. It sort of seems like one of those founder effects was to overinvest in technical research and underinvest in "social victory", i.e. convincing everyone to not build AGI. Whose fault was that? I think it was a distributed correlated failure, caused by deference. What should we have done instead?

One example of something we could have done differently would have been to be more open to the full spectrum of avenues, even if we personally don't feel like working on that / wouldn't be good at working on it / don't know how to evaluate whether it would work / are intuitively skeptical of it being doable. Another example would be to make it more clear when we are deferring to Yudkowsky or to "the community". We don't have to stop deferring, to avoid this correlated failure. We just have to say that we're deferring. That way, people keep hearing "I think X, mainly because Yudkowsky thinks X", and then they can react to "Yudkowsky thinks X" rather than "everyone thinks X" (and can check whether Yudkowsky actually believes X).

Currently most X-risk reduction resources are directed by a presumption that AGI is coming in less than a decade. I think this "consensus" is somewhat overconfident, and also somewhat unreal (i.e. it's less of a consensus than it seems). That's a very usual state of affairs, so I don't want to be too melodramatic about it, but it still has concrete bad effects. I wish people would say "I don't have additional clearly-expressible reasons to think AGI is coming very soon, that I'll defend in a debate, beyond that it seems like everyone else thinks that.". I also wish people would say "I'm actually mainly thinking that AGI is coming soon because thoughtleaders Alice and Bob say so.", if that's the case. Then I could critique Alice's and/or Bob's stated position, rather than taking potshots at an amorphous unaccountable ooze.

The open problem

There's a menagerie of questions we bump into in our lives. What food is safe to eat? Who should you vote for? What shape is the Earth? What effect would tariffs have on the economy? How easy is it to unify quantum mechanics and relativity? Was so-and-so generally honorable in zer private dealings? Which car rental service is good? How did https://wordpress.com/ come to be so good?? (Inkhaven brought to you by WordPress ❤️ .) What happened 50 years ago in Iran? What's happening right now in any place other than right where you are? Is genetic engineering moral? Will these socks wear out after 3 months? Should you get this vaccine? What's a reasonable price for a bike? Where should you hike? What's really going on at OpenAI? What is it dangerous to react sodium with? Is it legal to park here? When is it time to protest the government?

You can become an expert on almost any small set of these questions, such that you don't really need to defer very much to anyone else's testimony about them. But you can't become a simultaneous expert on most of the questions that you care about.

So, you have to defer to other people about many or most important questions. There are too many questions, and many important questions are complex and too hard to figure out on your own. Also, you can get by pretty well by deferring: a lot of other people have thought about those questions a lot, and often they can correctly tell you what's important to know.

But deference has several deep and important dangers.

If I'm not going to figure something out myself, how do I gracefully decay from a pure, individually witnessed understanding of the world (which was a fiction anyway), to a patchwork of half-understood pictures of the world copied imprecisely from a bunch of other people? How do we defer in a way that doesn't destroy our group epistemics, doesn't abdicate our proper responsibilities, properly informs others, coordinates on important norms and plans, and so on? How do we carve out a space for individual perspective-having without knocking out a bunch of load-bearing pillars of our ethics? How do we defer gracefully?

Discuss

Published on November 11, 2025 8:02 AM GMT

Do you ever write things that have to be under or over a specific number of words? Like right now I'm at Inkhaven where everyone has to publish a 500-word blog post every day by midnight or be kicked out. 

(Technically this doesn't include me — see the PS on my post from yesterday — but I want the true Inkhaven experience. Also there's candy in the Winner's Lounge, which you're allowed into after publishing your daily post. And technically-technically, the organizers say I'm allowed in there any time as a Contributing Writing, but, come on, I have some shred of pride. Not enough to keep me from counting this ridiculous parenthetical towards my word count, but a shred.)

Anyway, I was writing a new post here in the LessWrong editor and got sick of pasting my text into a Google Doc to check the word count. So, after a drawn-out fight with my robominions, I'm proud to say I have created a handy tool for seeing your word count in any browser-based editor. By which I mean the LessWrong editor and GitHub Issues, those being the ones I've tried.

I'm going to assume you don't care about the technical details (google "bookmarklet" if you do) and just give the instructions for getting this working:

1. Open my page of terrifying Javascript code, do select-all, and copy (control-a, control-c commonly).
2. Right-click the bookmark area below your browser's address bar and choose "Add page..." (Chrome) or "Add Bookmark..." (Firefox).
3. For the name, call it something like "wordcount" or "WC" if space is at a premium on your bookmark area.
4. For "Location" (Chrome) or "URL" (Firefox), paste the monstrosity from step 1. The newlines are automatically stripped out.
5. Hit Save.

That's it for setup. Now, while editing your LessWrong post or whatever, just click that bookmarklet. You'll see something like this pop up:

Except nothing will be red and crossed out like that yet. The idea is that there will be extra cruft on the page, like that "Untitled Draft — LessWrong" that shouldn't count towards your word count. So what you do is this: At the bottom of the popup, there's a button labeled "Copy exclusion tags". Click that and then hit paste at the bottom of your draft. You'll get something I can't show you without breaking the word count of my own draft here, because it uses magic strings. Ok, but I can show you a screenshot of what's at the bottom of my draft:

What is going on there? First is a note to myself to delete that stuff before before publishing. Fine. Then I have a copy of the title, since that totally counts towards my word count but my tool isn't seeing it unless it's in the body of the draft. Then comes the part where you tell it what words not to count. The fact that I have "Untitled Draft — LessWrong" in there is how the tool knows to exclude that from the word count. Which, oops, more meta problems. That string also appears legitimately here in this exposition but is now getting excluded:

It's less confusing when your draft isn't about this tool itself! 

The short version is, add the bookmarklet, click it while editing a draft, and any words that are being counted that shouldn't be, just stick a copy of them between those exclusion tags at the bottom of your draft. Also any words below those tags will be ignored, if you want a scratch area.

Ok, let me know if that's useful or if you find any bugs!

 

PS: To maximize the usefulness of this post, let me also include instructions for seeing your word count in Google Docs as well, though it's pretty self-explanatory there, comparatively speaking.

See "Word count" there in the Tools menu? That will pop up your word count and give you a checkbox, "Display word count while typing". 

Discuss

Published on November 11, 2025 7:50 AM GMT

I’d like to talk about a martial arts concept, which I think has some applications in rationality. It’s called stance, and you’ve probably heard of it.

I. Definitions

What is stance?

Stance is the way you stand and hold your body when you’re not at that particular moment doing anything, but you want to be ready to do something soon. There are many different stances for many different martial arts, but it’s not just a martial arts concept. Baseball pitchers have a stance when they’re on the plate. Massage therapists have a stance (they call it “posture”) when they’re standing next to the massage table. 

A stance is often useful because it’s something you can hold for a while if you want without putting too much strain on yourself, though often it takes a bit of practice to get into it. Instructors often have strong feelings about doing a stance correctly. The small adjustments to posture and angle might not seem important to a new student but often matter later, either when you hold that position for a long time or when you try to do more active movements from a bad stance. 

(Ironically, the obvious way to stand next to a table to give someone a massage will give you a crick in your neck after half an hour and this will get really bad if you keep using that bad stance five days a week, eight hours a day.) 

A good martial arts stance might make it quick and easy to block, kick, throw a punch, or suddenly turn and sprint for the door. There are tradeoffs; a good comfortable stance for a fencer is hard to move sideways from. That’s fine for fencing, which takes place on what’s essentially a narrow straight line, but wouldn’t be good for basketball where you might need to dart in any direction at a moment’s notice. 

And in martial arts (and in many fields) stance is something you’re trained to stay in unless you have a very good reason to be doing something else. It's not that you only are ever in one stance; many arts have several. It's not that you never leave stance; sometimes you do have a very good reason to be doing something else.

I think mental stances are useful as well.

II. Stance as a Default Action

When I find myself suddenly off balance, my first response is to go to a high horse stance.

Credit to The Tai Chi Notebook 

Why?

Well, if someone shoves me or if I overextend when kicking, I’m vulnerable to being thrown or knocked over. I don’t want that to happen, and my instructors didn’t either, so we drilled a habit of returning to a specific position that’s a little higher than in the picture. 

It’s a good place to be; I can move in most directions, I’m pretty well planted, but it’s the kind of thing I can hold for half an hour if I need to. (And believe me, I got plenty of practice on holding it for longer than that.) And it’s lower than I usually carry my weight, which means that if I was off balance then dropping into horse stance usually gets my back upright and ready. My head is up, where lots of ways I get shoved around or put off balance might mean my head and gaze is looking down at the ground instead of around at whatever’s going on.

If I don’t know what to do, go to high horse stance and look around.

I’ve dropped into it when slipping on ice, and it saved me from falling over. I’ve dropped into it when a rope bridge started swinging, and it helped me catch myself. And yes, I’ve used it when getting shoved. It’s a good neutral place to be in. It’s also a place I know how to get out of; I know how to roll from this, sprint from this, attack from this, all sorts of things. It’s not great to sprint from, but it’s not surprising or unfamiliar to me. You can kind of think of it as being step 2 of half the karate kata;[1] step 1 is get to horse stance, step 2 is be in horse stance, step 3 is doing basically anything else.

(That is such a horrible oversimplification I can hear my old sensei start to berate me. Sorry, to paraphrase an old maxim, writing about martial arts is a bit like dancing about architecture.)

The default nature of stance means that I don’t have to think about it. When things aren’t going well, if I don’t have a very specific plan I should get into stance and then reassess.

Have you ever been told to take deep breaths and count to ten when you realize you’re getting angry? That’s kind of a mental equivalent to this. It’s good to be in a familiar mental place from which you can make your next move.

III. Examples

What are some mental stances (or at least pointers to them) that you might already be familiar with?

• Take deep breaths and count to ten. Go into it if you notice yourself getting angry.
• The 3-2-1 method, where you name three things you see, two things you hear, and one thing you feel. Go into it if you notice yourself getting anxious.
• Challenging thoughts, asking “is this true?” when your brain suggests catastrophes. Go into it when you notice yourself thinking about how everything is awful.
• This meme:

Credit to Red Gold Sparks

I want to note that these are only sort of descriptions of the position. They’re more like instructions for how to get into the position, as though the way to refer to horse stance was to call it the feet-outside-the-shoulders-sink-at-the-knees position. They’re all pointed at getting you out of particular bad thought spaces and into something less off balance, more able to usefully react and assess the situation. 

During the Covid-19 pandemic, I worked from home. At first, I had my work desk next to my personal desk, which was next to my bed. This was bad for maintaining work-life balance. Eventually I moved my work desk to its own space, and I would maintain different music playlists as well as a small stretching routine before I started work. The goal was to keep a mental stance where I was focused and working, instead of the mental stance where I was trying to relax.

When I was on a sports team (quidditch, and yes, I know I'm a nerd) sometimes before a game the team captain would gather us into a huddle and talk about how hard we'd trained, how well we knew each other, and how we were going to leave it all on the field. We'd shout back that we were with him, that we were with the team, or just shout with enthusiasm. This is a very common way to get into a certain competitive mental stance, it's just one that I hadn't had much experience with before university. (Because, again, nerd.)

And then when I was losing sometimes I'd start to get angry or frantic. That was something I'd had experience with before, and I already knew it wasn't good, and I already knew how to recover from it. I was taught from a young age, as perhaps many of you were also taught, that it's okay to lose sometimes. It's a game. It's important to try hard, but it's more important to have fun. And I'd step into the mental stance where it's a game, and laugh at the feeling of springing off the ground under my feet and grin at the slap of the ball in my palms and relax into the simple joy of motion. "Don't be a sore loser" is, in its own way, a pointer towards a stance.

I've never liked being wrong. Saying the words "I was wrong" was hard growing up. I would have much rather given an oral presentation naked or jumped into ice water than admit I was wrong out loud. And yet, once I realized the importance of saying oops I realized I needed to practice, and practice I did. Fumbling blindly with only that distant candle, the stance I found was one where hiding the error is the more obviously foolish mistake. It feels fluid, a little unmoored, but now I have a stance where it doesn't hurt.

When I’m working on big projects with tight deadlines sometimes I feel like everything is out of control and I don’t even know what’s going sideways. That’s bad. My Project Manager horse stance is to start writing down every single task anyone is going to do for this project, then start tagging them with who is doing them and what priority the task is. It’s not just a useful artifact to have at the end, but also a position from which I can move whichever way I need to. If I’m confused and stressed and I don’t know what to do, getting into project manager stance is seldom the wrong move. Nothing I’m about to do in the next three minutes of the project is going to be made better by not having a list.

Even just reciting a mantra or catchphrase can be useful. Remember, some of the value of a stance isn’t just that it’s good, it’s that it’s a default path to get you somewhere that isn’t bad. It’s bad to be leaned so far forward you’re about to fall over, nothing in the next three seconds of the fight is going to be made better by being there. It’s bad to be so angry you’re hyperventilating, nothing in the next three seconds of the argument is going to be made better by being there.

Reciting "If the box contains a diamond" under my breath (or its close cousin, "metaphorical even bigger spiders Merrin") is a way for me to get in a particular mental stance where I'm ready to receive information that might be unpleasant.

IV. Usage

If there are different mental modes you occupy in life, especially if the habits you have from one mode would be bad to use in another mode, it can be good to think of them as different stances.

Even better is if you find some particular frame of mind effortful but useful. In those cases, drilling them as stances have paid off for me. It's easier to hold with deliberate practice.

I think it would be good and useful to practice mental stances. Pick one or two, and practice reliable ways to get into that headspace even when the unexpected happens.

I think it would be even better to have good mental stances and instructions for how to find them, other than the stance “calm.” I feel like there is more Art here to create, but I do not yet have more than the unwoven edges of it.

1. ^

   This is incorrect, I'm simplifying a lot here. For instance, I'm completely leaving out front stance, which is the other half![2]

2. ^

   Still simplifying.

Discuss

Published on November 11, 2025 7:00 AM GMT

In the film that the above still is from[1], the character sets his run-down hotel on fire, shoots some people, and then goes into his room to die.

I am confused about why this doesn't happen more often. That is, when people lose all their meaning in life (as must surely be happening regularly), why don't they explode? Personally I don't know anyone who's committed suicide, never mind had a nihilistic explosion where they burned down their house, or killed someone.

When people lose the meaning in their life, they lose a lot of the reason to keep putting energy into things. To believe in things. Being kind to their neighbor, supporting their friends, behaving well in public, holding down their job, eating well, exercising, sleeping enough... everything can become more effort than it's worth.

The world is a very big place, and I have heard stories of people losing all their meaning. Sometimes people become depressed; sometimes they move to a new country (e.g. from a developing country to a developed one) where they have nothing—no friends, no job, no community, and are very lost. A beta reader for this piece told me about someone they knew who, after an out-of-distribution dispiriting professional experience, had to move back in with their parents and get divorced; they talked about someone else who was cheated on in their life-long monogamous relationship since college, and was entirely unmoored. 

A man going through a nihilistic explosion.

I am reminded that it is typically not evolutionarily adaptive to be suicidal when things get bad. It's still worth it to keep striving, to keep putting in the work to try to find resources, mate, and rear offspring. Betting on it when things look lost, is much more "worth it" to the evolutionary forces, than for you to kill yourself and avoid the suffering.[2]

I think this is mirrored at the top level? There's no amount of resources you can gather or offspring you can create where your mind determines that it can stop aggressively tracking status, or to stop fearing death so much. Partly we're just made of stupid heuristics, and partly it just wouldn't be worth it for evolution to stop us from striving.

People talk of a hedonic treadmill, where as you run forward and get more strength and resources, your happiness meter stays roughly in the same place. But a treadmill is kind of unidirectional, so in my head it's more of a hedonic rubber band, or thermostat, that keeps you stuck in place whatever direction you go. People feel that losing a child is the worst thing that could possibly happen, and it is indeed horrendous, but most people pull themselves back together and get on with their lives.

Trapped in the hedonic rubber bandHave you considered using a freeze-ray?

From an abstract perspective, things can get very good, or get very bad. You can live in a flourishing era of prosperity and progress, or you can live in the USSR when millions die from unnecessary starvation.

But from the inside, with our hedonic rubber band, all of this is dampened. Everything feels more middling.

I think some people find this inspiring, that we'll always keep fighting? Or they take it as a reason to try to love the positive things more? I am suspicious that people don't use this argument to encourage people to commit suicide more often.

"Even a shitty, shitty life is worth living, apparently, because folks are living the fuck out of them."
—Louis CK

Similarly, things could get absolutely so much better, better than ever, and you won't really feel it. Have you imagined how good life can get? A few ideal romantic relationships, with incredible artists who understand you perfectly, and powerful hyper competent people who can wield the resources of the world as if by mere act of will, solutions to aging and a deeply fulfilling career in your chosen industry, always well slept and hyper healthy and feeling good in your body, pure being moving through intensely meaningful experiences in life... 

...now why don't you feel like that already? The dream has been reached for most humans in history—look how far child mortality has plummeted, look at how much food you have, look at how much great art you have at your fingertips, etc.

Mikhail Samin pointed out to me the other night that perhaps the true better way to live would be to temporarily commit suicide and freeze yourself cryogenically, in order to just live in the future (in expectation) where everything is finally good. If you could, wouldn't you rather bring people from the poverty of history to live in our prosperous world today? One can imagine a hedonic utilitarian supervillain going around, shooting his friends with a freeze-ray for their own good.

"Friends don't let friends live in the present."
—Harry James Potter-Evans-Verres (maybe)

All this is to say: your feelings are useful for tracking things in the world, but they will rarely let you feel how good or bad things actually are. People aren't making decisions in accordance with how things are, but with how things feel. And the feelings are dampened.

I think part of the mission of human rationality, is to better use one's feelings to track reality. I endeavor to learn to more naturally inhabit the full range of my possible emotions—to feel more horror, dismay, terror, and more delight, ecstasy, and tears of joy—to be able to track on an intuitive, gut, emotional level, whether things are good or bad. 

And then to use my System 2 and do math for the places that my feelings cannot go.

Whether to have kids in the USSR

There are two perspectives you can have, on people's feelings not tracking reality:

1. We should respect their feelings
2. We should not respect their feelings

These have different implications for natalism vs anti-natalism.

Suppose you live in a bad world. A world where things are horrendous, like you live in USSR that you can't escape and millions will starve to death, or a world developing AGI where everyone will likely go extinct in the next decade or two. Now, if their feelings should mirror the quality of the world they're in—"should", a word which here means "on reflection they will self-modify to be this sort of person"—then you're producing someone to primarily suffer excruciatingly and then consider killing themselves. It's quite plausibly something that, when they come in to their full powers, they would not have wanted to have existed.

Alternatively, if you respect their feelings within the hedonic rubber band, then it's kind of fine? They get to have a story, even if short. They get to love people and make friendships and help people in their suffering for a little while, in amongst all the pain and confusion.

I currently mostly don't respect my feelings, yet also think it's fine to have kids in the USSR. What's up with that?

My guess is that I want to be able to somehow get the best of both worlds—be emotionally quite functional in pretty bad worlds, while also being able to feel the full horror or bliss of whatever world I'm in. But is this attainable, or even rational? I'm not quite sure.

It's not quite the right tone, but I do like this GIF.

1. ^

   In rot13, the film is named Onegba Svax.

2. ^

   Some part of me would be equally able to explain that there's some fair decision theory that works out where you'll kill yourself when it looks really lost, and in-exchange put in a ton of extra effort when it's only nearly lost. Alas, the mind doesn't work this way. I suspect it's because evolution and brains are all pretty stupid and not able to consider this hypothesis or make this tradeoff.

Discuss

Published on November 11, 2025 6:50 AM GMT

Back in the day when debates about religion were fashionable, one of the standard back-and-forths went roughly like this…

Theist: If we reject God, then what’s to stop us from stealing and murdering each other?

Atheist: Well, mostly people don’t want to hurt other people that much, and also at the societal level we all benefit from laws against theft and murder and the like so we can just enforce those regardless of this whole “God” business.

Unusually Smart Theist: But low-trust societies are a thing. They have the laws against theft and murder, but they still suck. Empirically, a society just works way better if the large majority of its members subscribe to e.g. Christianity, and therefore generally don’t try to screw each other over even when they could get away with it.

Unusually Smart Atheist: Ok, but that doesn’t mean we need to accept e.g. the Bible as a package deal. It gets some things right, it gets some things wrong. But more importantly it’s a whole memetic egregore, and you can’t really get rid of the bad parts without rejecting the whole egregore. And you can get the good parts without most of the egregore; one certainly doesn’t need to claim that it was good and righteous for the ancient Israelites to kill every man, woman and child in Jericho, in order to get the parts we want.

[... and then the discussion would continue for several hundred rounds.]

Today, the religious egregores are not so dominant. But their niche is still filled by the memetic egregore Goodness - the egregore whose constituent memes are claims about what is Good.

The memetic egregore Goodness is the same type of thing as the older religious egregores. It feeds on the same feelings and instincts, and fills the same niche. And one can have basically the same arguments about it.

Goodist: If we reject Good, then what’s to stop us from stealing and murdering each other?

Agoodist: Well, mostly people don’t want to hurt other people that much, and also at the societal level we all benefit from laws against theft and murder and the like so we can just enforce those regardless of this whole “Good” business.

Unusually Smart Goodist: But low-trust societies are a thing. They have the laws against theft and murder, but they still suck. Empirically, a society just works way better if the large majority of its members subscribe to the modern Western memeplex of Goodness, and therefore generally don’t try to screw each other over even when they could get away with it.

Unusually Smart Agoodist: Ok, but that doesn’t mean we need to accept the memetic egregore Goodness as a package deal. It gets some things right, it gets some things wrong. But more importantly it’s a whole memetic egregore, and you can’t really get rid of the bad parts without rejecting the whole egregore. And you can get the good parts without most of the egregore; one certainly doesn’t need to e.g. buy into the Virtue Theory of Metabolism in order to get the parts we want.

[...]

Discuss

Published on November 11, 2025 5:49 AM GMT

Said Achmiz, citing Arthur Schopenhauer's The Art of Controversy, argues that debaters encountering an apparently crushing counterargument should not immediately concede:

Schopenhauer comments on people's vanity, irrationality, stubbornness, and tendency toward rationalization:

If human nature were not base, but thoroughly honourable, we should in every debate have no other aim than the discovery of truth; we should not in the least care whether the truth proved to be in favour of the opinion which we had begun by expressing, or of the opinion of our adversary. That we should regard as a matter of no moment, or, at any rate, of very secondary consequence; but, as things are, it is the main concern. Our innate vanity, which is particularly sensitive in reference to our intellectual powers, will not suffer us to allow that our first position was wrong and our adversary's right. The way out of this difficulty would be simply to take the trouble always to form a correct judgment. For this a man would have to think before he spoke. But, with most men, innate vanity is accompanied by loquacity and innate dishonesty. They speak before they think; and even though they may afterwards perceive that they are wrong, and that what they assert is false, they want it to seem the contrary. The interest in truth, which may be presumed to have been their only motive when they stated the proposition alleged to be true, now gives way to the interests of vanity: and so, for the sake of vanity, what is true must seem false, and what is false must seem true.

But, says Schopenhauer, these very tendencies may be turned around and harnessed to our service:

However, this very dishonesty, this persistence in a proposition which seems false even to ourselves, has something to be said for it. It often happens that we begin with the firm conviction of the truth of our statement; but our opponent's argument appears to refute it. Should we abandon our position at once, we may discover later on that we were right after all: the proof we offered was false, but nevertheless there was a proof for our statement which was true. The argument which would have been our salvation did not occur to us at the moment. Hence we make it a rule to attack a counter-argument, even though to all appearances it is true and forcible, in the belief that its truth is only superficial, and that in the course of the dispute another argument will occur to us by which we may upset it, or succeed in confirming the truth of our statement. In this way we are almost compelled to become dishonest; or, at any rate, the temptation to do so is very great. Thus it is that the weakness of our intellect and the perversity of our will lend each other mutual support; and that, generally, a disputant fights not for truth, but for his proposition, as though it were a battle pro aris et focis. He sets to work per fas et nefas; nay, as we have seen, he cannot easily do otherwise. As a rule, then, every man will insist on maintaining whatever he has said, even though for the moment he may consider it false or doubtful.

[emphasis mine]

Schopenhauer is saying that—to put it in modern terms—we do not have the capability to instantly evaluate all arguments put to us, to think in the moment through all their implications, to spot flaws, etc., and to perform exactly the correct update (or lack of update). So if we immediately admit that our interlocutor is right and we are wrong, as soon as this seems to be the case, then we can very easily be led into error!

So we don't do that. We defend our position, as it stands at the beginning. And then, after the dispute concludes, we can consider the matter at leisure, and quite possibly change our minds.

Schopenhauer further comments that, as far as the rules and "stratagems" of debate (which form the main part of the book)—

In following out the rules to this end, no respect should be paid to objective truth, because we usually do not know where the truth lies. As I have said, a man often does not himself know whether he is in the right or not; he often believes it, and is mistaken: both sides often believe it. Truth is in the depths. At the beginning of a contest each man believes, as a rule, that right is on his side; in the course of it, both become doubtful, and the truth is not determined or confirmed until the close.

(Note the parallel, here, to adversarial collaborations—and recall that in each of the collaborations in Scott [Alexander's] contest, both sides came out of the experience having moved closer to their opponent/collaborator's position, despite—or, perhaps, because of?—the process involving a full marshaling of arguments for their own initial view!)

So let us not demand—neither of our interlocutors, nor of ourselves—that a compelling argument be immediately accepted. It may well be that stubborn defense of one's starting position—combined with a willingness to reflect, after the dispute ends, and to change one's mind later—is a better path to truth.

Achmiz would later describe the behavior of being unwilling to budge to counterarguments as "normatively correct".

I opined that while I agree that it's possible to concede an argument too early and that good outcomes often result from being obstinate in the heat on an argument and reflecting at leisure later, I think describing obstinancy as "normatively correct" is taking it way too far.

In discussions of rationality, I usually take the word normative—of or relating to an authoritative standard—to refer to the authoritative standard of ideal Bayesian reasoning, often in contrast to what humans do. As Schopenhauer observes, the entire concept of adversarial debate is non-normative in this sense! Aumann's agreement theorem is a mathematical expression of Schopenhauer's dictum that "we should not in the least care whether the truth proved to be in favor of the opinion which we had begun by expressing, or of the opinion of our adversary." And yet people do.

Achmiz replied:

Indeed, so either we take this to mean that any normative claims about how to conduct such debates are necessarily meaningless, or else we allow for a concept of normativity that is not restricted to idealized Bayesian reasoners (which, I must remind you, are not actually real things that exist). Now, I am not saying that we should not identify an ideal and try to approach it asymptotically, but surely it makes no sense to behave as if we have already reached that ideal. And until we have (which seems unlikely to happen anytime soon or possibly ever), adversarial debate is a form of epistemic inquiry we will always have with us. So there must be right and wrong ways to go about doing it.

Retreating to a less censorious venue in which this discussion about social epistemology would be permitted to continue, I replied:

I emphatically agree that as long as there's a recognizably human "we", we will always have adversarial debate with us. Attempts to deny or disparage the necessity of debate in favor of some pipe dream of "collaborative truthseeking" have been and will continue to be a disastrous farce, like communism.

Nevertheless, when I consider why debate works, I don't want to use the word "normative" to describe the behavior of the obstinate debater, even while recognizing how the behavior plays a functional role within a larger system that discovers truths under the laws of normative reasoning.

In order to evaluate the truth of a proposition, a cognitive system must evaluate the evidence (or arguments) for and against it. It turns out that humans are particularly adept at rationalization, latching on to a proposition that we want to be true (respectively false), and searching hard for arguments for (respectively against) it. As a result, it turns out to be incredibly productive to pit the efforts of a human who wants the proposition to be true against the efforts of another human who wants the proposition to be false. Driven by vanity to maintain whatever they have said, the disputants will thoroughly explore the space of arguments concerning the proposition, and each painstakingly correct his adversary's errors—probably more thoroughly and more painstakingly than if they were merely curious about the proposition.

Debate works—more than anything else. As long as people are human, it could hardly be otherwise. To simply declare that "we should in every debate have no other aim than the discovery of truth" is about as actionable as delcaring that the economy should function "from each according to his ability, to each according to his need"—and with similar results.

And yet, all this having been said acknowledging that the success of adversarial debate has deep roots in human nature, I think it's important to notice the ways in which human nature is normatively silly. Yeah, okay, I see how it is "that the weakness of our intellect and the perversity of our will lend each other mutual support"—that the need to defend our position gives us motivation to keep working on the argument rather than prematurely folding, and that this isn't even particularly bad for the cause of Truth in the world given that the other guy is going to be doing it, too.

But the perversity of our will is still perverse! That's the opposite of normative!

Sometimes the adversariality of debate is formalized, as in court trials. The plantiff and defendant's counsel each have their functional roles to play within the system, and that role is not to be fully general truthseekers. (That's closer to the judge and jury's job.) In that context, we can say that it is normatively correct that a lawyer should argue for her client's interests, rather than to disinterestedly persue the truth of the case.

But in the absence of such formalized roles, where it's understood that the individual is to be a cog in a larger system—no, it is not normatively correct to refuse to budge in response to counterarguments! When I'm having a discussion, and my interlocutor makes a point against my "side"—or if I notice it independently—I want to acknowledge it right then and there. It's not a natural impulse, but I think it's learnable, and worth learning, as a step towards a better world.

Because—normatively—there shouldn't be "sides". Or if there should be sides, that means the interaction is a conflict rather than a disagreement, and it would be normatively correct to reveal that fact, rather than maintaining confusion about it.

Another commenter suggested that when faced with an obstinate interlocutor, one should troll them with, "But you know I'm right; you just don't admit it."

Achmiz said:

To which the obvious answer would be, "I know no such thing. I might—or might not—know that your argument seems like it doesn't have any obvious problems. But, as we all know, you can prove anything with apparently-not-obviously-flawed arguments. So who knows if you're right or not? My epistemic state right now is ‘I am right, just like I thought before we started arguing'. If that changes, I'll let you know."

I said:

To which the obvious answer would be, "You're lying about your epistemic state. You're not completely epistemically learnedly helpless: if you really thought that reason was so perfectly impotent that apparently-not-obviously-flawed arguments could prove anything with equal ease, you'd never learn anything at all. In order for your belief that you were right before we started arguing to mean anything, it needs to make probabilistic predictions. As long as reason isn't perfectly impotent, that will include predictions about intellectual discussions: being right implies being quantitatively less likely to see people come up with counterarguments that you don't know how to answer than if you're wrong."

Achmiz replied:

Nah, I don't buy that at all. That sort of view suggests a model of reasoning where we can do precise Bayesian calculations on everything all the time (which of course we can't) and where, furthermore, we can perform all updates and think through all consequences near-instantly (which of course we definitely can't).

There is, after all, a difference between "I see no flaws in your argument, right now" and "I see no flaws in your argument, after having thought about it for a while, and looked at the matter from all angles at my leisure, and taken as much time and effort as I think is productive to take to think through all relevant considerations". Your view would only make sense if there were no such thing as computational uncertainty or bounded rationality—if our brains were infinite computers! But they aren't.

So if you make an argument which, apparently to me, proves some point, what is that? A piece of evidence. I will go and consider it, and perhaps I will conclude that I should update my views by some amount in some direction. (Perhaps in the direction of your views, perhaps not.) But that's all.

As for "being right implies being quantitatively less likely to see people come up with counterarguments that you don't know how to answer than if you're wrong", I am not even sure that's true. If it is true, what is the odds ratio? How could we possibly establish it? Merely being right doesn't seem to me to predict the absence of counterarguments I can't easily and immediately answer.

Finally, attend closely to what I wrote (describing my hypothetical response to an apparently convincing argument): not "I have made no update whatsoever, however miniscule", but "I am right, just like I thought before we started arguing". This is perfectly compatible with your view! I could easily say: "Yep, I do indeed think that your apparently convincing argument is a very, very small amount of evidence for your position. Yes, apparently-not-obviously-flawed arguments can't prove anything with equal ease; yes, I expect it to be quantitatively less likely that I should see people come up with counterarguments I can't easily and immediately answer if I am right than if I am wrong. All of that is true, I have taken it all into account; I have made the appropriate update, which is very small; and of course I still think that I am right. Maybe I'll make a larger update later, once I have considered everything carefully, and the status of your argument changes from ‘there are no immediately obvious flaws' to ‘yeah, on consideration, this really is right'. Or maybe I won't! We'll see!"

I granted that one shouldn't update much at first if the process of evaluating arguments is naturally slow, but elaborated on why I thought the lack of a ready response was informative:

Before you start arguing with someone, you have some prior idea of how people are likely to respond. (Hopefully because your opinion is not completely uninformed, such that you've read previous relevant discussions and are familiar with the standard arguments on the topic, but more fundamentally because you can't not have a prior.)

When the responses come in, you should have a sense of them being stronger or weaker relative to your expectations. To some quantitative-in-principle (if small and certainly not quantitative-for-humans) extent, the responses being weaker than you expected should give you confidence, and the responses being stronger than you expected should give you pause.

Restated with more concreteness, if the responses are the same nonsense you've already refuted a dozen times before, but coming from otherwise sharp people, that's a sign that you're right (because if there were sensible counterarguments to what you had said, you'd expect these otherwise sharp people to find them).

If the responses are novel challenges that would take you hours to come up with a non-embarrassing counterreply to, and coming from Reddit-tier riff-raff, that's a sign that you're wrong (because if there weren't sensible counterarguments to what you had said, why are these clowns putting on such a good show of it?).

As you point out, if evaluating arguments just naturally takes a long time, then it's a weak sign. But what's been at issue in this thread is whether it's "normatively correct" "to stick one's heels in and be unwilling to budge on a position regardless of reason or argument." I'm saying: no, it's not; the common psychological phenomenon Schopenhauer is describing in which the weakness of our intellect and the perversity of our will lend each other mutual support is normatively incorrect, because we should not in the least care whether the truth proved to be in favor of the opinion which we had begun by expressing, or of the opinion of our adversary.

It makes sense to need time to think things through, but I think it's tendentious to call that stubbornness in the context of the preceding discussion. "Let me think about it and get back to you" is not an expression of being "unwilling to budge".

Achmiz replied:

The problem with this argument is that "let me think about it and get back to you" is useless for continuing a conversation. You say something, I say "let me think about it and get back to you", and... then what? We pause for weeks or months or who knows how long? Then I come back and I say "ok, I thought about it for a while and I still think I'm right". You say "ok, in that case ..." and you make another argument. Or I make an argument and you have to think about it. Pause for unknown amounts of time again. We can't have any kind of discussion like this.

Even if this were somehow epistemically optimal for you and for me (a hypothetical pair of arguers), what of any audience? Or third, fourth, fifth participants? Does it remotely make sense to proceed like this? No, of course not.

What's more, by always doing things in this way, you miss out on the (very likely! very common in practice!) possibility that after many back-and-forths, a long sequence of argument and debate wherein you say many things and I say many things and we each do not budge from our starting positions and do not immediately make any updates on the basis of apparently good arguments from the other, we then both go away and think for a while, and read other things, and talk to other people, and—armed with the many things each person's respective interlocutor said—reconsider our respective positions. If after the first apparently convincing argument made by one of us, the other person had said "let me think about it and get back to you", this would not have been possible.

And on top of all of that, in the latter scenario we would deprive all audience members of the benefit of having an adherent of both of our respective positions represent the complete position, for full exploration and examination, in the course of a temporally compact debate (rather than one which takes place over who knows how long while each participants goes away and thinks after every apparently good argument by his interlocutor). (And this isn't even touching on the practical question of how likely it is to have any given debate participant actually be available after the ostensible thinking time. People get busy, distracted, dead...)

All of this means that (as I say in the linked LessWrong comment) while "let me think about it and get back to you" is appropriate sometimes, it is often entirely correct to continue to hold a position in a debate, even when an apparently convincing argument against it has been made. Certainly I will think about the matter at my leisure, and I may change my mind (and I may even change my mind in such a way as to bring me closer to your view!). But for the purposes of the debate we're having, I will not immediately update, and this is indeed normatively correct. (There are exceptions to this, because it is possible, e.g., to make an argument such that your interlocutor acknowledges that you've made a good point, which he basically already agrees with, but just wasn't thinking about, hadn't recalled, etc.)

I said:

The point of "Let me think about it" is that you shouldn't be continuing a conversation if you don't have anything honest to say. If you do have more points to make that haven't already been covered in previous remarks, then by all means, you should continue the full exploration and examination of arguments.

But if an apparently convincing argument has been made, then you don't have a counterargument ready (on that particular point, which may be a small part of a wide-ranging discussion). That's what it means to be apparently convincing. Isn't it better to be honest about that, than to bluff and bluster in order to maintain your position; or to ghost in order to maintain ambiguity about whether the argument was so apparently convincing that you need to think about it, or you just have other duties to attend to?

[Regarding depriving the audience of an adhrent representing each position,] the benefit is in the "full exploration and examination" part, not the "adherent of both of our respective positions" part; it doesn't matter who contributes which parts of the examination. If you make a flawed but repairable argument, and I'm a philosopher pursuing truth rather than a lawyer protecting my client's interests within an adversarial system, it's normatively correct for me to fix it for you if I can.

Achmiz replied:

It's certainly fine (indeed, good) to be honest about the fact that you don't have a counterargument ready on some particular point. However, that's no cause to grant to your interlocutor the validity of the position he's arguing for. Why should you, when you haven't had time to fully consider the matter? "I don't currently have a reply to that point, and I am not (at this time) updating in the slightest on that basis, which is the correct move on my part, because I genuinely don't know whether you are in fact correct, and my current lack of a counterargument doesn't license me to conclude that you are correct" is a perfectly honest stance.

(It's not like "apparently convincing argument that, upon reflection, turns out to be total bullshit" is some sort of esoteric species! On the contrary, it's quite common to encounter such things.)

[Regarding the claim of the benefit being in the "full exploration and examination" part, not the "adherent of both of our respective positions" part:] This seems obviously wrong. In most cases, an adherent of a position will represent it best, both due to familiarity and due to interest. There are of course exceptions, but they generally imply a large mismatch in the arguing parties' intelligence, knowledge, etc. I definitely prefer to see a view explained and defended by its adherents, rather than only by its detractors. Don't you?

[Regarding the the claim about a philosopher fixing their interlocutor's repairable arguments:] If! But why should I assume that you're "a philosopher pursuing truth"? Even if I think that you're attempting to be such, why in the world would I assume that you're succeeding? Isn't this just the old nonsense about "we're all members of a truth-tracking community and should treat each other accordingly"?

In general, I find your whole position here to be inconsistent. You seem to be appealing to a hypothetical scenario where everyone involved is a perfectly honest and perfectly rational philosopher earnestly seeking truth, etc., on the grounds that this would be normative, therefore it is normative to behave in a way that would be appropriate to such a scenario. But given that this scenario is entirely counterfactual, why would it be a good thing to act as if it obtains?

I clarified:

I said, if I'm a philosopher pursuing truth, it's normatively correct for me to fix your broken arguments (as part of the full-exploration-and-examination-of-arguments endeavor). Your trust is neither required nor relevant. [...] [Y]ou can improve your adherence to the ideal unilaterally. It doesn't matter whether the other guy is less rational or less honest. When I admit that he's made a dent in my initial position, I'm not doing it for his sake.

Achmiz didn't understand:

For whose sake are you doing it, then? Can't be for your own—that's unaffected by what you say in a public debate. Is it for the sake of the audience (or other participants)? But I claim that you actually do those people a disservice by doing this. (Indeed, you even do yourself a disservice by doing it!)

I articulated a toy model:

Suppose Reality is either A or B, each with prior probability 0.5. Reality generates 20 cards [...] e1.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} , e2, ... e20.

I look at e1, e2 and e3, and they all say "A". That's three information-theoretic bits of evidence that reality is A. That's 8:1 odds, or probability 8/9 ≈ 0.89 that Reality is A.

I encounter you claiming that Reality is B with probability 0.94. "Nonsense!" I cry. By way of explaining, I show you e1 through e3.

You show me e17, e18, e19, and e20, all of which say "B".

My response should depend on how it came to be that you showed me e17, e18, e19, and e20.

If I were certain that you were a psychic all-seeing perfectly dogmatic B-partisan that got to look at all twenty cards and are showing me all of the cards that say "B", then I can infer that all of e4 through e16 say "A" (because if any of them didn't, you would have shown me them). I should end up with posterior odds of 216−4=212=4096:1, or probability about 0.9998 that Reality is A.

If I were certain that you're a philosopher pursuing truth just showing me all the cards you happened to find, then I should count your cards the same as mine and have posterior odds of 23−4=1:2, or probability 1/3 that Reality is A.

Realistically, I expect that you're definitely not all-seeing, and probably some sort of B-partisan but not perfectly dogmatic about it, so my posterior probability should end up somewhere in between the two extreme cases. It would depend on the details. It's not worth specifying the details, because this is just a toy formalism meant to illustrate a qualitative principle of normative behavior. I'm not claiming that real-world arguments are literally isomorphic to cards representing a known likelihood ratio; I just think doing the arithmetic for the cute story about cards is helpful for disciplining the mind and preparing it to confront the normative principle which many find hard to face.

And that principle is, just be honest. That's it! Just be honest! If the card says "B", update on it (where, again, the correct update depends on how you think evidence is being filtered). If the argument seems convincing, say so. (And if you don't see an error now, but you think you likely will on further consideration, then it shouldn't seem "convincing"; you need to recalibrate your sense of convincingness until it's not making predictable mistakes like that.)

I don't think this is a complicated idea. It's not—well, I won't say "it's not hard", because that would be a lie. Schopenhauer knew.

But crucially, as I've been saying, you can be honest unilaterally. The reason debate is crucial for human beings and the "we're all members of a truth-tracking community and should treat each other accordingly" hokum is something only con artists say is because most arguments aren't honest: people have some belief they want to get on the shared map for reasons other than its truth and selectively search for "A" or "B" cards to reveal. There's a fear that if you admit it when the other guy has a point, that puts you at a disadvantage in the competition to get your preferred beliefs onto the shared map (such that even an ideal philosopher who somehow didn't have any motives other than making the shared map more accurate shouldn't do it).

And it would put the philosopher at a disadvantage if the audience can't tell the difference between honesty and dishonesty: if you show all your cards, and the other guy only shows his "B" cards, and everyone treats your revealed cards and his equally, then that would incorrectly bias the shared map towards representing Reality as B.

But I think people can often tell? If I don't "make it a rule to attack a counterargument, even though to all appearances it is true and forcible", that makes it more meaningful when I attack a counterargument that doesn't appear true and forceful, if people are tracking my behavior over time and can infer that I'm not bluffing.

Achmiz wasn't buying it:

I agree that you should be honest. You should honestly say "I have no reply to that at this time, and I will absolutely not budge at all in my position, because I consider my inability to come up with a reply at this time to be uninformative, and to tell me essentially nothing about whether your argument actually is valid or is instead nonsense, or mistaken, or has some obvious counter which I am not thinking of at the moment". (Or something to that general effect. Certainly you shouldn't lie and say "actually I think that your argument is wrong", or something.)

That would be honest. And correct! Normatively correct!

On the other hand, it would be incorrect to say "hm, I find your argument convincing, so I am updating in the direction of your position". No! Wrong and bad! Very common among rationalists to do this, and totally not the right way to do things.

(Whether it would also be dishonest to say something like that depends on whether you understand that it's incorrect, of course.)

You say: "the correct update depends on how you think evidence is being filtered". Well, how should I know how the evidence is being filtered? Trying to guess how the evidence is being filtered is how they get you. (For a stark example of this failure mode, see Scott Alexander's habit of going "I assume that [the government / the media / third-world dictators / criminals / Sam Altman ] is not telling me the whole truth, but I am going to assume that I can guess approximately how much they're lying, and compensate for their lies, deducing the truth thereby". It doesn't work!) I decline to do this. This is how you get got.

And that's easy mode, even. What about when (as is the vastly more common case) we are not talking about some quantitative "evidence", which you can do math on (including the chance and degree of filtration), but rather an argument, a model, etc.? This is the real problem, you see—not physical uncertainty (where you don't know what is the probability distribution across various possibilities, whether those be on the object level, or on the meta level of "how is this evidence being filtered"), but computational uncertainty, where you can't, in the moment, consider all the implications of your interlocutor's argument, can't fully analyze it, etc. The argument seems convincing, but we know perfectly well that this might just mean that you can't spot the problem right away. Why should you change your view on the basis of "seems convincing"? You shouldn't. It should take "I've given it much thought, and I am actually convinced" to change your mind.

I confessed:

My problem with [honestly saying that I have no reply but will absolutely not budge at all] is that I can't honestly say that, because I don't consider my inability to come up with a reply to be uninformative. Why would it be uninformative?

Often times in such situations I do still think I'm ultimately right for reasons I have yet to satisfactorily articulate. I could say that (and expect to do more work and probably come back later with a satsifactory reply). But I couldn't say "I will absolutely not budge at all in my position" with a straight face. The fact that I have more work to do, and that I can't be surpremely confident in the soundness of work I haven't yet done, seems like a form of probabilistic budging.

Achmiz had the last word:

Why would it be informative?

(I mean, sometimes it might be. But sometimes not.)

One question to ask in such cases is "informative to whom"? To the ideal Bayesian reasoner who has unlimited computing power and thinks infinitely fast? Yeah, maybe. To you, personally (or to me, etc.)? I really don't see why it must be informative.

But maybe we've gone too long without any actual examples (a sign of this being that we seem to be going in circles). So let's try and make this more concrete. Consider some scenarios:

Scenario 1

I make some argument. Someone makes a counterpoint. The counterpoint is one which I hadn't thought of, but once I see it made, I immediately see that it's correct and that it severely undermines my argument and basically I am just wrong. I am immediately convinced. I no longer think that my previous position was correct. If I had thought of the counterargument myself, I would never have made the original argument. I've actually changed my mind. I say "oh yeah true, good point, you are right".

(Example.)

Scenario 2

I have some position, which I am quite sure is right. I am very sure that my view is well-considered and accounts for all possible challenges. I make an argument for my view. Someone makes a counterargument, and it's one which I hadn't anticipated at all. If you had asked me what my interlocutor might say in response to my argument, I would not have predicted anything at all like the response I've received. Their counterargument seems sound. I have canned replies to many kinds of potential rebuttals, but this one just comes out of left field. I have literally never considered anything like this counterargument I've just heard. I am shaken. Obviously, I have no reply to make. I am not convinced of my interlocutor's view (I'd have to do a lot more thinking before that could become the case), but my confidence in my own view has just taken a serious blow.

(Examples: many cases of someone who's been raised or indoctrinated to believe in a highly dogmatic system, then encounters a real adherent of a competing system; the latter is familiar with the former's beliefs, but not vice-versa. Religion, Marxism, etc.)

Scenario 3

I have some position, which I am quite sure is right. I am very sure that my view is well-considered and accounts for almost all possible challenges, and while I'm aware of some potential reasons to be slightly less than absolutely certain of my belief, nevertheless I have put considerable effort into examining my view, considering challenges to it, etc. I am also aware that there exist many specious and nonsensical arguments against this view, and I am reasonably familiar with the broad categories thereof, and have invested effort into carefully considering many (though not all) of them. The "evidentiary crossfire" of all the positive reasons to be confident that my view is right, and the lack of any really compelling reasons to doubt, adds up to a high degree of confidence.

Someone then makes an argument against my view. I have not encountered this specific argument before, but it resembles various other arguments I've seen, and adds basically nothing new to my understanding. I have no immediate reply to make, but hearing this argument doesn't surprise me at all. I already knew that there are many of these bad arguments out there, and having the answers to every single one of them ready to hand simply does not strike me as a good use of my time. I do intend to reply to this one, but not at once. My confidence is not shaken at all.

(Examples: religion—from the other side—is the obvious one, but also, this essentially parallels the structure of Jaynes's "resurrection of dead hypotheses" scenarios, such as the ESP example he gives.)

Scenario 4

There exist many bad arguments for various completely wrong and extremely stupid ideas, which I have no interest in spending all my time rebutting. I encounter one such argument for one such idea, to which I have no ready reply. I shrug. It means nothing. Arguments? You can prove anything with arguments.

(Examples: basically everything that Scott Alexander talks about in his classic essay on "epistemic learned helplessness".)

[Regarding not being able to claim to be unmoved:] One distinction I think it's useful to make is between "you are convinced" and "you think that you should be convinced".

Many people don't make this distinction. It is, I've found, a skill—indeed, one may call it a rationalist skill—to distinguish between "I have heard arguments for X, and I don't know of any rebuttal" (or any number of more or less related epistemic states, like "I've been told X, by someone who I hold in esteem"), and "I actually, for real, think X" (a.k.a., simply, "X"). (Luria's peasants had this skill, though! And it might also be said that this is nothing more than Hanson's "far mode vs. near mode"...)

Scott's hypothetical engineer who thinks "oh, there's a good argument for terrorism; I guess I should become a terrorist" lacks this skill. Many "rationalists", in my experience, also lack it. (This does much to account for the persistent popularity of utilitarianist moral views.)

So, someone makes an argument, which seems to have no obvious flaws; you have no immediate reply. Alright. The one comes to you and suggests that you should therefore be convinced of the argument's conclusion, and should update your own views accordingly.

But are you convinced? Are you, actually, in fact, convinced? Do you actually now believe that your previously expressed position is mistaken? Contrast my scenario 1 with scenario 3: in #1, I am convinced! Actually, genuinely, I now believe differently than I did before! I do not need to ask myself if I should be convinced; I simply am convinced. In #3... not so much.

(One may argue, here, that the presence of bias means that the question "are you actually convinced" is unreliable as a guide to truth. Yes, perhaps, but note that this does nothing to rescue the alternative. If you have biases which prevent you from being actually convinced when a perfectly unbiased reasoner would, in your place, be convinced, then you should of course try to overcome those biases and rectify your epistemic outlook, but it is foolish in the extreme to say "I am biased, therefore I will simply pretend that I am convinced whenever it seems like I should be convinced". That's how you end up on the proverbial train to crazytown!)

What I am advocating is to not pretend that you're convinced. If you're convinced, fine. If you're not, then you're not. Maybe because you're in scenario #3 or #4, maybe because you haven't had a chance to think, maybe who knows why. This is right and proper.

Discuss

Published on November 11, 2025 5:25 AM GMT

Context: Every Sunday I write a mini-essay about an operating principle of Lightcone Infrastructure that I want to remind my team about. I've been doing this for about 3 months, so we have about 12 mini essays. This is the first in a sequence I will add to daily with slightly polished versions of these essays.

The first principle, and the one that stands before everything else, is to question the requirements.

Here's how Musk describes that principle:

Question every requirement. Each should come with the name of the person who made it. You should never accept a requirement that came from a department, such as legal ... you need to know the name of the real person who made the requirement.

Then, you should question it, no matter how smart that person is. Requirements from smart people are the most dangerous, because people are less likely to question them. Always do so, even if the requirement comes from me [Musk]. Then make the requirements less dumb. 

Here's some of how I think about it: plans are made of smaller plans, inside their steps to achieve 'em. And smaller plans have lesser plans, and so ad infinitum. 

But it's hard to get your subplans right up front.

There are a bunch of reasons for this: sometimes something looks like it might be easy from 1,000 feet, and a lot gnarlier at 10 feet. Sometimes, one of your subproblems looks like it has a standard solution, but that standard solution is built to solve a bunch of issues you don't care about. The point of questioning the requirement is to de-silo your subplan, and look at a larger scope, and use that to figure out a better plan.

Most work done in most organizations is work that is best done never at all. The core of most solutions turns out to be much more elegant and simple than anyone who worked on the first version of something was able to imagine: 

Evolution of the SpaceX Raptor engines. Source

Why do we have requirements at all? Of course, we have plans and subplans, but those don't constitute "requirements". As will be a recurring theme in these principles, the need for things like the "requirements", and the associated risks, rear their head when a project needs more than one person to be completed, and as such, when tasks need to be split up across many individuals, and often shudder hierarchies of individuals. 

The best way I have found to set up a delegee to understand the goal of a task and to question the associated requirements, is to communicate two things at the same time:

1. A solution sketch: How you would approach this problem if you were starting to work on this? Do not shy away from giving details at this point. Talk about specific libraries, or tools you would use, or specific marketing phrases. Be as concrete as possible, especially about what you would do to get started.
2. The goal: What is the broader problem this task fits into? How did you notice this problem? Whose problem are we solving? Why is it important? 

Abstractly communicating goals is usually a doomed endeavor, so start with a concrete solution sketch. Then, taking the solution sketch as a starting point, explain what problems it is trying to solve. No plan survives contact with the enemy, but the initial plan is still usually the best tool you have for conveying your eventual goal.

As a delegee, question the requirements. Throw away the solution sketch you were given, and think about how you would go about this problem from first principles, in the context of its broader goal.

In conversation this looks like this:

Weekly memo: Robert's priority this week is to set ourselves up to use the caching tools we now have available on Next.js to make a bunch of our page responses faster

Robert: I question the requirement to use Next.js's caching tools for this. I agree we might want a caching layer, but their API is kind of horrendous and has unstable in the name. IMO we should just set up our own Redis instance if we want to do more caching. 

Me: Oh, yeah, sure, that works as well. Actually... now that I am looking more into this, we maybe want to wait for a bit on the next major version release which does something totally different for caching. 

Robert:  I am not finding myself enthused about their proposed approach, but agree it looks better than running our own Redis. I'll see how fast I can upgrade us to the latest version.

Most requirements come in the form of uncommunicated assumptions, so the above conversational pattern often plays out at different levels of abstraction: 

Weekly memo: Ronny's priority this week is to simplify our check-in system since it's been a giant pain for the last 3 events that we ran, and I think we could just roll something ourselves with the Unifi API and Airtable that's much smoother. Feel free to ask Rafe for help on the technical parts. 

Ronny: Ok, but my current take is that we shouldn't have a check-in system at all. We made like $3k with individual room-bookings during the last few events, and that was already sacrificing a bunch of rooms that could have been session spaces. I think we might want to cut the number of rooms we make bookable by 60%, and then I don't think the effort of optimizing the room booking system is worth it. 

Tomorrow: "Do not hand off what you cannot pick up"

Discuss

Published on November 11, 2025 5:09 AM GMT

First part of a series of article on French AI Policy that I’m currently writing as part of the Inkhaven Residency.

For three centuries, France has stood among the great powers of the world, and there it wants to stay. Now, far from the heyday of La Belle Époque, where the French Empire stood shoulder to shoulder with the British Empire, France has resisted the pressures to align too closely with the world superpowers, lest it become just a vassal state. French pride mandates that French sovereignty must be preserved.

As World War II came to an end, the two blocs of the Cold War emerged. Despite the pressures to align, France was not willing to trust the US security guarantees. It needed a seat at the table. France secured its own permanent seat at the UN security council in 1945, its own nuclear weapons in 1960, and its energy independence through nuclear power in the 70s, even withdrawing from part of NATO in 1966. It seems that history may have proven them right; as Trump hints that the US might not come to defend Europe, France stands secure with its own nuclear deterrence force, even suggesting France might become Europe’s nuclear umbrella.

This independence from the US gave France the freedom to pursue its national interests, even when they conflicted with US priorities. The most prominent case being the 2003 Iraq war, where France decided to not support the US proposal at the UN Security Council, to the stupefaction of American foreign policy experts. Even this year, France went against the US position and decided to recognize the Palestinian State ahead of the UN General Assembly. Closer to home, France has regularly contested proposed EU policies that would put its sovereignty at risk, like the proposed EU-wide arms procurement, which allow spending EU money on US weapons.

The importance of sovereignty is key to understanding France’s positions on AI. French elites have correctly understood that AI is becoming one of the most strategically important resources of this century, and they have seen how the US has already started weaponizing its dominance of the AI supply chain for its policy objectives.

This has led France to invest heavily into securing its access to compute, both through building a sovereign semiconductor supply chain in Europe, and through €100B+ of investment in AI datacenters build-out, including a partnership with the UAE to build a 1GW datacenter.

On the algorithmic side, France has chosen Mistral as its national champion, which it has supported both through direct investments, and through attacking the provisions of the EU AI Act targeting systemic risks from general purpose AI. The open-source strategy is not a whim, it’s a key part of Mistral’s value proposition, an AI provider that is European, sovereign, and taking a stand against the power centralization of American Big Tech.

Through continuous investment into its sovereignty, France has remained a strong and independent live player. While it may key its usual alignment with US positions, it will not accept a US hegemony over the AI era. France has proved time and time again that it will stand alone if it needs to.

Discuss

Published on November 11, 2025 5:09 AM GMT

In Rudyard Kipling’s “Wee Willie Winkie,” Winkie is a six-year-old British boy and the son of a Colonel posted in colonial India. His highest ideal to become an honorable man. He strives to be just, prudent, and loyal, in the ways a six-year-old believes these things can exist as true, real things. Not as means for some other end, but as ends in themselves. He lives with his whole heart, and he has a six-year-old’s lisp, and it’s easy to fall in love with him in just a couple thousand words.

By promising to keep a soldier’s engagement secret, he finds himself with a feeling of responsibility for that soldier’s betrothed. In the climax he sees her foibleing into danger, and rushes after her to help. By the time he catches up to her they are deep in enemy territory, and soon surrounded by one of the clans that are unhappy with the British presence. They’re about to be kidnapped and likely killed.

Winkie orders the raiders to bring word to the British outpost that they need help, and promises them they’ll be rewarded. The raiders laugh at first, until one of them recognizes the boy.

“He is the heart’s heart of those white troops. For the sake of peace let them go both, for if he be taken, the regiment will break loose and gut the valley. Our villages are in the valley, and we shall not escape. That regiment are devils. They broke Khoda Yar’s breast-bone with kicks when he tried to take the rifles; and if we touch this child they will fire and rape and plunder for a month, till nothing remains. Better to send a man back to take the message and get a reward. I say that this child is their God, and that they will spare none of us, nor our women, if we harm him.”

 

Canonically, Harry Potter was protected from the Killing Curse because his mother loved him so darn much. It’s a common fantasy trope, to the point of being a cliche. I read another protected-by-love and I roll my eyes. Usually.

Here, however, a helpless boy is protected by the power of the love of the local regiment of soldiers. The threat of them boiling over in murderous vengeance is a shield more effective than a dozen rifles and a cannon. The soldiers have created the real magic that these wish-fulfillment fantasy stories dream of. With their love. And the raiders know this, because they also understand love.

The power of love is that you don’t have to make any threat, it is inherent in the display of love. Those who are being deterred don’t need to judge how serious you are or what other political or practical considerations may sway you, they only need to be aware of the depth of your love. The Afghan raiders in Wee Willie Winkie don’t need to weigh the political situation of the British outpost and how retaliation will affect their strategic position in the wider area. They just need to know that the soldiers there absolutely adore Winkie and will rage like a thousand suns if he’s killed, all other consequences be damned. This is the shield that protects the boy.

In the modern world this is unacceptable. To say that I feel it’s good that these soldiers would raze an entire valley if their Winkie was killed is borderline psychopathic. But my heart feels this is good anyway. I don’t think you can have love without this drive to smash egregious violence into the bodies of anyone who would kill your loved one.1 I don’t think it’s good to pretend otherwise. Recognizing that you would hesitate to go on a vengeance rampage is a sign that you aren’t truly in love with the person you’re with. Maybe people avoid looking at that because realizing they aren’t in love with their partner would be very inconvenient.

In fact by strangling this desire in ourselves and burying it deep inside, we may be damaging our ability to feel true love at all. When your body isn’t allowed to feel this drive to do violence for the memory of your loved one, it doubts you love that person at all. Instead of love you get a warmed-over Liking. Maybe you even Like Like someone. But love? You can’t isolate the love from the willingness to do violence. They come as a pair.

Discuss

Published on November 11, 2025 4:40 AM GMT

(Crossposted from my Substack; written as part of the Halfhaven virtual blogging camp.)

She wasn’t ready, but the man started speaking. “Hello, Ms. Tatsuo. My name is Ethan Blande from the Public Health Agency of Canada. I wanted to ask you some questions because I’m told you were the first to report symptoms.” The man was dressed in a suit and wore and N95 mask over his stubble.

Ai nodded, but didn’t speak. She was still short of breath from having gone to the washroom connected to her hospital room. At least she was still able to go by herself. Some of the other patients had come into the hospital later than her, but had already turned fully grey.

“At what time did the symptoms start? How long before you came into the hospital?”

“About… ten minutes before,” gasped Ai. “I came… right away.”

“Okay,” said Ethan, typing a note on his laptop, which rested on the adjustable overbed table. “And what were the initial symptoms?”

“I first noticed… the grey eyes. Even my irises…” The man waited for her to continue, but seemed impatient. “Then I heard… the voices. Muffled… I still can’t hear what they… what they’re saying.”

“One patient described the voices like an alien radio station. Do the voices seem to be human voices, or something else?” asked Ethan.

Ai felt anger in her chest. Why was this man asking her stupid questions? Was he as clueless as everyone on the internet? She hoped the Public Health Agency would have some answers, but apparently not, if they were chasing alien stories. “Sound human… to me,” said Ai firmly.

The man wanted to ask her more questions, but one of the nurses, a stern older woman named Violeta, ushered the man away so Ai could rest for a while. She gave Ai a puff of a bronchodilator — albuterol — using an inhaler. Ai noted the dose and the timing in a spreadsheet on her phone. She’d been tracking every drug she’d taken since becoming pregnant. That was doubly important now that she was sick.

A while after her breathing returned to normal, Shirley, a chipper health care aide, brought a tray into the room. Her pink uniform carried an assortment of pastel-coloured baubles, including a little, ineffectual pair of pink scissors. Her hair was done up with a big bow holding it together, like Minnie Mouse.

“Ding ding ding! Good afternoon, Ai! I hope you like burgers!” She set the tray down on the table. “Oh, you have some blood, let me clean you up!” she chirped, wiping blood from Ai’s mouth. Ai stared blankly at the lifeless hospital burger.

“I always feel like burgers have too much bread, y’know?,” said Shirley. “But another patient came up with a brilliant idea! If you throw away the bottom bun, you get more of the condiments and meat in each bite. I call it an ‘urger’. Because you’re missing part of the burger…” Shirley’s voice trailed off when she realized Ai was glaring at her. “Okay, just let me know if you need anything! Toodles!” Shirley skipped her way out of the room in her pink crocs. Ai would have rolled her eyes if she weren’t so exhausted. Instead, she started watching the news on the hospital room TV, picking at her burger without really eating it. She hoped the pain in her belly wasn’t coming from her womb.

Ai sipped apple juice as she watched the TV in the corner. No matter the time of day, the news was about her disease. The “Greyscale Disease” — so called because its victims looked like they were straight out of a black and white film — had infected over 800 people in Toronto and thousands elsewhere in the last six days. Experts on the news argued about the nature of the disease. It didn’t seem to be a bacterium or virus. Some said it could be a prion or parasite, but the nasally man on the screen was insisting it was an environmental contaminant.

People in more advanced stages of the disease had lost limbs, but nobody had died so far. That was the one saving grace. Nobody yet knew if Greyscale was lethal. Ai herself hadn’t lost any limbs or fingers, but the disease continued to get worse. Her entire body was grey, now. And unless she slowly sipped apple juice, there was an ever-present taste of blood in her mouth. She had even bled from her eyes. The doctors wanted to give her drugs to control the bleeding, but it wasn’t profuse, and nobody knew what the effects of such drugs could be on a mystery disease.

Shirley peeked into the room and let Ai know it was time to take her pills.

“I’m not taking them! You can tell that to Dr. Wahid!” Ai shouted after her, but the woman was already gone. “I don’t need painkillers, I need DMSA!” She had read about some patients with Greyscale online claiming to have improved their colour after taking dimercaprol, and DMSA worked similarly to filter toxins from the blood, but was safer for pregnant women.

“Hey!” Ai shouted, getting out of bed. She felt lightheaded, and pain spiked in her belly, but she was determined not to let Shirley get away. It could be hours before she saw the doctor again unless she sent Shirley to fetch her.

Other patients might be content to wait for the lazy hospital staff to do their job, but if you wanted better outcomes than other people, you had to be more cautious than other people, and in the hospital, that meant being proactive.

She stepped out of her room and tried to yell after Shirley, but instead felt liquid spill from her mouth. The ever-present voices in her head grew louder. She suddenly felt faint.

The hospital was crowded with Greyscale patients, so they were doubling up on rooms. Ai had hoped the new patient would at least be able to keep her company, but the man — an elderly man named Gill — was in pain and wouldn’t respond to anyone except to moan and occasionally shout for more pain meds. He had been hit hard by the disease. He lost an arm and a foot. They didn’t fall off, but seemed to shrivel back into his body. And though he looked underweight, he was unnaturally heavy. It had taken three health care aides to lift him from the wheelchair into his bed. Ai could feel the same heaviness in her bones. It wasn’t just weakness. The Greyscale patients were getting physically heavier somehow, in apparent defiance of physics.

When Ai looked at him, she saw her own future, which filled her with terror for herself and her baby. She knew there must be someone, somewhere who had found some kind of treatment to at least slow the disease, and so she spent her time searching news articles, journals, and even reddit looking for anything that could help.

There were many stories about the progression of the disease, which only served to scare her further. First, the voices and the grey skin. Then bleeding, shortness of breath, and heaviness. Hair starts retracting into the body. Then more bleeding. By that point, even the blood is grey. Fingers start retracting into the body. Eyes and noses, sometimes. Limbs. Still, nobody had died.

Equally useless were the conspiracy theories. People saying Greyscale was an attack from China, or some kind of alien first contact gone awry. No doubt the voices contributed to that theory. Ai’s own voices were louder now, but still indistinct, like someone talking in a shrill voice. Ai couldn’t make out any actual words.

Most promising were the many stories of treatments working, though most were quickly debunked. Still, people online recommended all manner of treatments: deferoxamine, an experimental peptide, a medicinal herb, or a prayer. Ai was skeptical of them all and feared taking random treatments without knowing how they could affect the disease, but cataloged them all in a spreadsheet. She would take anything if it would protect her baby, but random treatments could just as easily make her condition worse.

Still, every few hours she would grow desperate and demand Shirley get her the doctor. She would ask for this or that drug or supplement, and the doctor usually refused. When they did oblige, she often didn’t take the pills, having changed her mind. She was obsessed with protecting her baby, but paralyzed by fear of doing something wrong. Somehow, she couldn’t shake the idea that she’d already done something wrong. That if she’d only eaten healthier or exercised more, maybe she wouldn’t have gotten the disease in the first place.

Despite her obsession, she continued to get worse, and the pain in her belly grew.

Something hit Ai in the head. “Ma’am, wake up!” Gill croaked.

Ai looked over at him, then at the floor. He’d thrown a shoe at her. Before she could protest, he nodded excitedly toward the TV. “Look!”

There was a breaking news report. “Cure for Greyscale Found.” The newscaster announced that a small company from Texas called Chronic Systems had reversed the symptoms of one patient in Texas. The unlikely company was a physics research company, rather than a pharmaceutical one. Ai dismissed it as yet another fake cure, but apparently the CDC was taking it seriously and said they were rushing to have the treatment available to patients within the next 72 hours.

Gill looked at her with eyebrows raised. He was hard to look at, as the disease had taken one of his eyes. Rather than disappoint him with her skepticism, Ai just shrugged.

The Public Health Agency of Canada was working with the American CDC to get the treatment to every Greyscale patient. The treatment Ai now held in her hands. It was a small metal device shaped like a pill, suspended inside a specialized glass vial stamped with the name “Chronic Systems”. The pill was apparently somewhat radioactive, and had to be transported in these protective vials. She turned it in her hands, ruminating.

Gill had taken the treatment two hours ago. If she wasn’t mistaken, his colour was improving. But she couldn’t be sure. Even without the treatment, Greyscale patients sometimes had their symptoms improve temporarily, only to get worse later. It was a risk, and she wanted to wait and see if it worked for other patients before taking it herself. She had a baby to worry about. But if she waited too long, that might be just as bad.

She felt wetness between her legs and knew she’d just lost control of her bowels again. It would be mostly blood. Grey blood. She was mortified. Just in time, the obnoxious health care aide Shirley skipped into the room. “Don’t worry,” she said, “I’ll get you cleaned up!”

Ai rolled to her side and tensed as Shirley changed her diaper and cleaned the mess. The disposable wipes were cool on her colourless skin. The chemical smell wiped away the metallic smell of her bloody, grey feces.

“There you go!” The woman’s happy voice was like nails on a chalkboard. “Let me know if you need anything else, okay?”

Ai stayed on her side. “Just leave me alone.”

Ai woke in the middle of the night, feeling the urge to defecate again. This time, she managed to hold it. Hoping not to make another mess for Shirley, she got out of bed. Her heavy feet hit the ground with a thud. She grasped her walker. One of her fingers had shriveled into her hand, but she didn’t have time to dwell on it.

As quickly as she could, she shuffled to the toilet. She felt lightheaded, and her breathing was heavy. She noticed her shadow seemed to follow her with a delay, which made her uneasy. She made it in time, though her belly screamed with pain. The voices seemed louder than ever, though she still couldn’t understand what they were saying.

She was concerned with the amount of blood coming from her. Maybe she should have been taking the medications to help with bleeding. Figuring out what was safe and what wasn’t was impossible, and she couldn’t trust the doctors to do it. Not when medical errors were one of the leading causes of death.

As she stood up from the toilet, her head swam. She shuffled to the sink, where she realized she’d left the vial with the metal pill. She didn’t remember bringing it into the bathroom with her.

In the mirror was a healthy Ai, with normal skin and long black hair. Not the grey and bald woman she really was. The Ai in the mirror was smiling. Happy. The reflection of the vial in the mirror was empty. She picked up the vial to examine the metal pill inside. Her reflection did the same, though only after a short delay.

Ai started trembling with fear. Something was wrong. Was she hallucinating? The incessant voices in her head were deafening, but still, she couldn’t understand what they were saying.

She turned to call for help, but there was a stabbing pain in her stomach. A kick, she realized. From the baby.

Suddenly, she was able to place the voices. They weren’t coming from her head at all, but from her womb. Muffled through all the flesh, it was her baby, begging her. Screaming at her to take the pill.

Ai’s shaking hands dropped the vial, which shattered on the tile floor. The metal pill came to rest between glass shards. She bent down to pick it up. The effort made her gasp for breath.

She swallowed the pill. She nearly choked on it, but she got it down. She lie down on the ground to catch her panting breath, not caring about the shards of glass cutting into her calves. The pill was inside her now. Maybe it was a horrible mistake, or maybe she’d just saved her life. The uncertainty was like a long, black night.

The cure worked. Ai felt nearly back to her normal self after a few days, aside from the missing finger, which would never return. Her baby was healthy, too. Gill had already gone home earlier that day, and she would be going home in a few hours.

She knew she should be relieved. But she was not. It had all been so random. She couldn’t point to a time when she had made a mistake, or done anything right. She had behaved essentially at random, and it all worked out for no reason.

Another breaking news broadcast was starting just as Shirley hopped into the room. She was wearing scrubs featuring Tweety Bird in a repeating pattern. “Breaking news? What’s that?”

“I don’t know,” said Ai, hoping Shirley wouldn’t keep talking over the news report.

The news report showed the CEO of Chronic Systems in handcuffs. Ai’s heart sank. Was something wrong with the cure?

The newscaster spoke. “Bharat Ashwin, CEO of Chornic Systems, arrested just a few hours ago in connection with the Greyscale Disease epidemic. Experts from the CDC say his company’s illegal physics research may have been the original source of the disease which his company later cured. We go now to Ethan Blande from the Public Health Agency of Canada for comment.”

Ai looked at Shirley, whose eyes were wide, and mouth clearly agape under her mask. As usual, the woman’s reactions to everything were exaggerated and unnecessary.

Ethan Blande appeared on the screen. “We are working with the American FBI in the arrest of Mr. Ashwin. The discovery was made after one of our analysts noticed that not a single patient refused treatment in Canada. We reached out to the CDC and found the same thing was true for American patients. This is unheard of for a patient population this size. Further investigation into the company’s activities revealed that they had been studying exotic time physics, and internal documents relating to their so-called cure showed it contained something called ‘temporal antibodies’. The company’s own documents indicate they believe the disease itself to have been caused by the cure, even though the cure is taken after the disease already sets in. Without their so-called miracle cure, the disease itself would never have occurred.”

“What does that mean?” asked Shirley. “The cure fixed the disease in the present, but went back and time and made you sick in the first place?”

“That’s what he said. Can’t you listen?” snapped Ai. But Shirley was right. It didn’t make any sense.

“Thank you Mr. Blande,” said the newscaster. “Bharat Ashwin is being charged with criminal negligence, though his lawyer claims that Mr. Ashwin could not have forseen or avoided this unprecedented ‘closed causal loop’, as he calls it. Mr. Ashwin himself declined to comment. Just a few minutes ago, Michio Kaku tweeted about this unprecedented-“

Shirley turned off the TV. “That’s enough of that. Are you ready for your breakfast?” Just like that, the woman was ready to move past a revelation that shook Ai’s conception of reality.

“How can you just start talking about breakfast, just like that? Did you hear what he said?” Ai grilled. “Do you have any conception of what he just said? I was sick with a disease that went back in time! The cure for the disease is what made me sick! If I had resisted taking the cure, would I never have gotten sick in the first place? Or would I have always taken the cure, no matter how hard I tried not to? This makes no sense! Is any of it even true? How does this not bother you? Are you too vapid, too empty-headed to understand? Do you just not care?”

Shirley stood frozen for a minute, then let out a heavy sigh. “Ai, did you have any of the other health care aides wash you while I wasn’t working? They were rough, right? Like they didn’t care?”

Ai nodded. She was right. Shirley did a good job, but some of the others were like barbarians, rushing so they could get to the next patient. Leaving her not feeling fully clean.

“I take my time. I like to make sure people are clean. And happy, if possible. Sometimes I meet some really nice people here. And sometimes those people die. Then I go home and I cry. But the next day, I come back here wearing my Tweety Bird scrubs and I try to put a smile on people’s faces. Do you know why?”

Ai was feeling guilty, and spoke softly. “To make patients more comfortable?”

“Yes, but it’s not just for them, it’s for me, too. You can’t control life, but you can choose your attitude. You can choose to smile in the face of a hurricane. You can say to life, ‘bring it on!’” Shirley balled her hand into fists, and seemed to be challenging the universe itself.

Ai felt ashamed for insulting her. Maybe wasn’t as shallow as she seemed. Maybe Shirley had some things figured out that Ai herself still needed to work on. Especially if she wanted to do a good job raising her baby. Ai looked down at her belly. “I hope I’ll be a good mother.”

“You will!” Shirley assured her. “I can tell you care. That’s the most important thing. Now, how about I get you and your baby,” she patted Ai’s belly, “some breakfast?”

Ai nodded. “Okay.”

“I can’t hear you!”

“Okay!” Ai said more loudly, and smiled.

Discuss

Published on November 11, 2025 4:11 AM GMT

Here is an excerpt:

America appears to be, at the moment, in a sort of benevolent hostage situation. AI-related spending now contributes more to the nation’s GDP growth than all consumer spending combined, and by another calculation, those AI expenditures accounted for 92 percent of GDP growth during the first half of 2025. Since the launch of ChatGPT, in late 2022, the tech industry has gone from making up 22 percent of the value in the S&P 500 to roughly one-third. Just yesterday, Meta, Microsoft, and Alphabet all reported substantial quarterly-revenue growth, and Reuters reported that OpenAI is planning to go public perhaps as soon as next year at a value of up to $1 trillion—which would be one of the largest IPOs in history. (An OpenAI spokesperson told Reuters, “An IPO is not our focus, so we could not possibly have set a date”; OpenAI and The Atlantic have a corporate partnership.)

Many people believe that growth will only continue. “We’re gonna need stadiums full of electricians, heavy equipment operators, ironworkers, HVAC technicians,” Dwarkesh Patel and Romeo Dean, AI-industry analysts, wrote recently. Large-scale data-center build-outs may already be reshaping America’s energy systems. OpenAI has announced that it intends to build at least 30 gigawatts’ worth of data centers—more power than all of New England requires on even the hottest day—and CEO Sam Altman has said he’d eventually like to build a gigawatt of AI infrastructure every week. Other major tech firms have similar ambitions.

Listen to the AI crowd talk enough, and you’ll get a sense that we may be on the cusp of an infrastructure boom. And yet, something strange is happening to the economy. Even as tech stocks have skyrocketed since 2022, the companies’ share of net profits from S&P 500 companies has hardly budged. Job openings have fallen despite a roaring stock market, 22 states are in or near a recession, and despite data centers propping up the construction industry, U.S. manufacturing is in decline.

It’s clear that AI is both drowning out and obscuring other stories about the wobbling American economy. That’s a concern. But even worse: What if AI’s promise for American business proves to be a mirage? What happens then?

The yawning gap between data-center expenditures and the rest of the economy has caused whispers of bubble to rise to a chorus. A growing number of financial and industry analysts have pointed out the enormous divergence between the historic investments in AI and the tech’s relatively modest revenues. For instance, according to The Information, OpenAI likely made $4 billion last year but lost $5 billion (making the idea of a $1 trillion IPO valuation that much more staggering). From July through September, Microsoft’s investments in OpenAI resulted in losses totaling more than $3 billion. For that same time period, Meta reported rapidly growing costs due to its AI investments, spooking investors and sending its stock down 9 percent.

Much is in flux. Chatbots and AI chips are getting more efficient almost by the day, while the business case for deploying generative-AI tools remains shaky. A recent report from McKinsey found that nearly 80 percent of companies using AI discovered that the technology had no significant impact on their bottom line. Meanwhile, nobody can say, beyond a few years, just how many more data centers Silicon Valley will need. 

…

Boom and bust can feel like two sides of the same coin: Consider also that if AI companies deliver on their massive investments, it would likely mean producing a technology so capable and revolutionary that it wipes out countless jobs and sends an unprecedented shock wave through the global economy before humans have time to adapt. (Perhaps we will be unable to adapt at all.) If they fail, there will likely be unprecedented financial turmoil as well.

The biggest lesson of the past two decades of Silicon Valley is that Meta, Amazon, and Google—and even the newer AI labs such as OpenAI—have remade our world and have become unfathomably rich for it, all while being mostly oblivious or uninterested in the fallout. They have chased growth and scale at all costs, and largely, they’ve won. The data-center build-out is the ultimate culmination of that chase: the pursuit of scale for scale itself. In all scenarios, the outcome seems only to be real, painful disruption for the rest of us.

Discuss

Published on November 11, 2025 2:49 AM GMT

People have been celebrating Secular Solstice for over a decade now, in our small community. Many different programs and versions have been collected at Secular Solstice Resources (and elsewhere). The amount of material can be overwhelming. Many Solstice programs are based around original material being written or updated, speeches that are specific to the speaker, and other things that make it challenging to reuse.

The goal for this program is to be an easy-to-follow, easy-to-reuse Solstice program for a group celebrating Solstice for the first time, or the first of a few times, possibly a small group, possibly without many resources.

Simply print one copy of the program per participant (or fewer and have people share), and follow the directions.

If you aren't familiar with the songs, you will need one or a few people to go through them first and help lead. You could host a listen-through party first to get your group familiar with the songs used. NOTE: I haven't made a specific playlist for this yet, due to time constraints, but reference materials for all the songs can be found at Secular Solstice Resources' Songs page. (Please express interest in the comments if you'd like this and I will try to make time!)

>>Just give me the printout already!<<

Additional editorial notes about this Solstice:

• I have focused on using the most commonly-reused songs and speeches from Solstices past. See appendix.
• I have focused on using material that is less specific to a particular time and place than others, though near-everything in Solstice relies at minimum on factual claims that may need to be updated from time to time. It is a value of our community to update in the face of evidence, so if this gets too wrong or out of date and I don't fix it, please don't use material you regard as wrong!
• Where possible within other constraints, I have tried to use songs that are easier to sing along to and/or produce.
• I wrote an original Introduction to Solstice speech that should hopefully be generic enough to cover most cases-- as always with Solstice, please do edit (or ask me to make changes) if you see inaccuracies, and feel free to customize if you would like.

Appendix: Most common Solstice songs and speeches

I wrote a script to extract these from the programs on Secular Solstice Resources. Here are the results, as of January 2025:

Most used songs:

SongUsesBrighter_Than_Today38Uplift33Bitter_Wind_Blown22Hymn_to_the_Breaking_Strain21Time_Wrote_the_Rocks19Here_Comes_the_Sun17Five_Thousand_Years17When_I_Die16Always_Look_on_the_Bright_Side16X_Days_of_X_Risk15Bitter_Wind_March14Chasing_Patterns14Do_You_Realize13Bold_Orion12Still_Alive12Endless_Light11Here_and_Now10The_Sun_Is_A_Mass_Of_Incandescent_Gas10Blowin_in_the_Wind8Somebody_Will8Lean_on_Me8Voicing_of_Fear8

Most used speeches:

SpeechUsesMinute_of_Silence17Beyond_the_Reach13Road_to_Wisdom10500_Million_But_Not_A_Single_One_More10Pale_Blue_Dot10Gift_We_Give_Tommorow8Call_and_Response_Defiance_Abridged8We_Are_Here7This_Is_a_Dawn6Litany_of_Tarski6No_Royal_Road5Origin_of_Stories_Morning_Edition4You_Cant_Save_Them_All3Communal_Meal3Toasts_Boasts_and_Oaths3Story_of_Winter3The_Goddess_of_Everything_Else_Abridged3Origin_of_Stories_Twilight_Edition3Call_and_Response_Defiance3Only_Human3

Discuss

Published on November 11, 2025 2:26 AM GMT

Many prominent figures, including Sam Altman and Elon Musk, have suggested universal basic income (UBI) as a solution when artificial intelligence renders human labor obsolete. Musk has even promised "universal high income." However, there are serious reasons to be skeptical of this vision. The framing assumes that alignment will be solved but the AIs are aligned to some elite group or they uphold anything like current capitalism with its property rights.

from @MarioNawfalThe Trust Problem

Elon Musk has said that "in the benign scenario, probably none of us will have a job" and promises not just UBI but "Universal High Income." However, when he took a government role with DOGE, one of his first moves was to defund foreign aid programs including PEPFAR (which provides HIV treatment), as well as programs fighting malaria and tuberculosis. A government memo estimated the cuts could cause millions of additional deaths, with one analysis projecting approximately 600,000 deaths, two thirds of the deaths being children. Bill Gates remarked that "the picture of the world's richest man killing the world's poorest children is not a pretty one." While some of these cuts were later reversed, infectious disease programs for HIV are still reduced to about 30% of pre-cut funding levels. 

The Leverage Problem

Your work is your leverage in society. Throughout history, even the worst rulers depended on their subjects for labor and resources. Mutual dependence is why society functions: why you can go to the supermarket and buy food, why housing exists for you, why society is "aligned" to provide you with space to live a decent life. And if you are more useful to others than the average person, you get rewarded proportionally. In a fully automated world, you become a net burden. If we assume that the police and military have also been automated, there is no realistic option to rebel or protest for change either. Resource dependent economies may give us an early glimpse at this, many resource rich countries have an extremely poor population and no democratic institutions.

@the_yancoThe Information Problem

If everyone is unemployed, that means all media is automated too. All the information you consume would be AI-generated and AI-curated. We're already seeing how AI-powered social media algorithms fragment society. These systems, currently optimized merely for engagement, have discovered that lies, misinformation, and conspiracy theories often generate the most interaction.

Now imagine a world where not just the algorithms but all the content itself is AI-generated. Imagine algorithms explicitly designed to mislead and divide. People couldn't organize their thoughts coherently enough to even conceive of rebellion or imagine alternative social structures. 

Caveat: We Probably Won't Get That Far

It's extremely difficult to imagine a society where we're all unemployed and humanity is still around. Think about what's required for a mass unemployment scenario: an AGI system powerful enough to run society, automate all jobs including research, police and military. Such a system would obviously be lethally dangerous, therefore if we get it catastrophically wrong, we die. We only get one shot at this and without a promising plan the odds don’t look good.

The Bottom Line

Being useful keeps us alive. This is more fundamental than constitutional rights or free speech. In the real world, we're aligned with other humans largely because we need each other. Our jobs mean we're still useful to the economy, useful enough that others will provide us with the goods and services they produce.

But more fundamentally: we're unlikely to survive long enough to face this dilemma. The kind of AGI powerful enough to automate all jobs is the kind of AGI powerful enough to end humanity if misaligned. And we're rushing toward building it with no credible plan for alignment. 

Discuss

Published on November 11, 2025 2:19 AM GMT

My post on the grapefruit-juice effect contains a ternary plot of citrus fruits. Here it is again (source):

Ternary plots are great! They're used in a number of specialized fields, but I think they would be more widely popular if people were more familiar and comfortable with them. Let's make it happen.

Look at the citrus fruit plot. Every data point on that plot corresponds to a species of citrus, whose ancestry is some mixture of mandarins, pomelos, and citrons. The three corners are the ancestral species themselves, which are 100% themselves and 0% anything else. To take an example from somewhere inside the plot, most lemons are about 20% pomelo, 30% mandarin, and 50% citron. You can read that off from the grid lines: mandarin ancestry is shown by vertical position, with 0%-mandarin fruits along the bottom of the triangle and the mandarin itself at the top. The "lemons" cluster is about 30% of the way up (between the yellow horizontal lines marked "20%" and "40%"), so it's 30% mandarin. Similarly, lemons are 20% of the way from the right edge (0% pomelo) to the bottom-left corner (100% pomelo).

Note that there are three variables (percent pomelo/mandarin/citron ancestry), but they are not independent; they always sum to 100%, so there are only two independent "degrees of freedom". This is why you can make a two-dimensional plot of the three-dimensional data.

The one confusing thing about ternary plots is that the axis lines are usually only marked on one edge. For example, the horizontal lines marking mandarin ancestry are labeled on the right-hand side; on the left, you instead have angled labels in blue, which correspond to the sloped lines for pomelo ancestry. Most ternary plots don't use different colors in the helpful way this one does; I think the best way to read them is to look at a corner (100% of something), identify the opposite edge (0% of that thing), find the lines parallel to that edge, and read off the labels on whichever side of the triangle is oriented to match those lines. (If the labels aren't helpfully rotated to match up with the lines, you can instead go with whichever side reaches 100% at the corner and 0% at the side in question.)

Use case

You can (and should) use a ternary plot to display any kind of quantitative data with:

1. Three numerical components to each data point
2. All components non-negative (positive or zero)
3. A constraint on the sum of the three components

Actually, if you instead have a constraint on a different linear combination of the components, you can still make (something like) a ternary plot; it just won't be an equilateral triangle. If the second rule (non-negativity) doesn't apply you can still make a ternary plot, but some points will fall outside the triangle, so you'll have to extend your grid lines.

The rest of this post is fun examples.

Mixing diagrams

Most ternary diagrams are for mixtures of things, where the sum constraint corresponds to the fact that percentages have to add to 100%. The citrus fruit diagram is one example.

This mixing diagram for soil has gone somewhat viral:

Clay, silt, and sand have a technical definition in terms of grain size. Below .05mm, sand becomes silt; below .002mm, silt becomes clay. So this plot gives a sort of low-dimensional projection of the infinite-dimensional space of granular mixtures. In fact, you can think of an arbitrary granular mixture as having a sort of spectrum of grain sizes, described by a spectral density function, and this plot shows a two-dimensional quotient of the vector space of those functions...

Speaking of spectra, one can also make a ternary plot for RGB colors! Here it is, embedded in the broader space of all possible colors:

The colors outside the triangle are too saturated to be displayed on a standard computer screen. The horseshoe shape is a little harder to explain; maybe in a future post.

This plot shows the conditions for a mixture of methane, nitrogen, and oxygen to be flammable. (Note that the grid-line labels are oriented unhelpfully, so you have to be careful about using the correct scale when reading off percentages.)

And here's one for frequencies of different alleles.

Other ternary plots

QAPF diagrams are basically two mixing triangles glued together:

Sums and differences of the electronegativities of a pair of atoms can be used to construct a sort of ternary diagram of bond types, called a van Arkel--Ketalaar triangle:

(Hmm. One could apply that trick to quite a lot of things...)

The baryon decuplet is often drawn in a sort of ternary diagram of quark content -- although the chart slightly predates the idea of quarks:

(source)

Actually, there's some deep math behind that: the triangular lattice appears here as the root lattice of the Lie group A2.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} , also known as SU(3). In most derivations, the lattice itself shows up as a diagonal slice through a 3D Cartesian grid; the lattice points are all the points with integer coordinates that sum to zero.

Ternary plots are also used in astrophysics, for non-gaussian statistics (measurements involving a triangle of three points in the sky at once). Unfortunately, these are usually displayed with two axes for ratios of edge lengths of the triangle, instead of as a ternary plot of the three angles, which sum to 180 degrees. Also, they're Fourier-transformed.

Discuss

Published on November 11, 2025 2:14 AM GMT

How large of a breakthrough is necessary for dangerous AI?

In order to cause a catastrophe, an AI system would need to be very competent at agentic tasks[1]. The best metric of general agentic capabilities is METR’s time horizon. The time horizon measures the length of well-specified software tasks AI systems can do, and is grounded in human baselines, which means AI performance can be closely compared to human performance.

Causing a catastrophe[2] is very difficult. It would likely take many decades, or even centuries, of skilled human labor. Let’s use one year of human labor as a lower bound on how difficult it is. This means that AI systems will need to at least have a time horizon of one work-year (2000 hours) in order to cause a catastrophe.

Current AIs have a time horizon of 2 hours, which means it’s 1000x lower than the time horizon necessary to cause a catastrophe. This presents a pretty large buffer.

Currently, the time horizon is doubling roughly every half-year. That means that a 1000x increase would take roughly 5 years at the current rate of progress. So, in order for AI to reach a time horizon of 1 work-year within the next 6 months, it would mean that the rate of AI progress would have to increase by 10x, leading to a doubling in time horizon roughly every two weeks instead of every 6 months.

AI breakthroughs of the recent past

It seems like a huge breakthrough is necessary to make 5 years of AI progress happen in less than 6 months. Has a breakthrough of this size ever occurred?

I can mainly think of two recent examples of AI breakthroughs that might be comparable in size to what’s needed to create dangerous AI in the short-term.

Case 1: Transformers

One is the invention of the transformer architecture. We don’t have time horizon estimates for any models before GPT-2, as even our easiest tasks are probably too difficult for earlier models. However, we can maybe get a sense of how many years of progress it is through other means.

Epoch AI estimates that transformers represented a compute efficiency gain of 10x-50x. According to Epoch AI, the historical rate of effective compute increases is around 3x per year, and algorithmic advances have accounted for roughly ⅓ of AI progress in LLMs since 2014. That means that transformers represent a 9-15 month jump in AI progress[3]. This places it way below the 5-year threshold required to get current models to a dangerous time horizon.

Also, the algorithmic progress from transformers wasn’t instant. It took two years to go from the Attention is All You Need paper to large transformers like GPT-2 being released. So it’s likely that the invention of transformers is neither large enough, nor sudden enough, that a similar invention would be very dangerous if it happened tomorrow.

In fact, if we round the impact of transformers to a 1-year jump in AI progress, then we’d need five transformer-sized breakthroughs compressed in a 6-month timespan to reach a 1-year time horizon.

Case 2: AlphaFold

The other example is AlphaFold. While AlphaFold is not a general AI architecture, it’s still useful for establishing an upper bound of how crazy breakthroughs can get.

I haven’t seen a good analysis that tries to answer the question “How many years of protein-folding progress did AlphaFold represent?” But judging from this analysis, it seems like it’s at least 5 years, and maybe in the decades.

This means that AlphaFold is an existence proof for at least 5 years of narrow AI progress being made in a short period of time.

What is the probability of 1-year time horizons in the next 6 months?

Assuming transformer-sized breakthroughs happen every 10 years, and events are independent, then the probability of five such breakthroughs happening in the next 6 months is very, very small.

But AI breakthroughs aren’t independent events. They have the same inputs, and the occurrence of one breakthrough is an update towards the inputs being sufficiently high to produce additional breakthroughs quicker. Additionally, AI breakthroughs could feed into each other, as the “AI capabilities” output of AI R&D can feed back into the “intellectual labor” and “compute spend” inputs.

Narrowly superhuman AI leading to generally competent AI

So, if multiple transformer-like breakthroughs are unlikely to lead to AIs with 1-year time horizons, what about AlphaFold-like breakthroughs? If we think that there could be a narrow AlphaFold-sized breakthrough in coding, then it becomes plausible that AI R&D could be automated in the short-term.

For any verifiable domain that AI researchers are trying to “crack”, I’d guess that AlphaFold-sized breakthroughs are less than 5% likely per year. And programming seems harder to “crack” than protein folding, as evidenced by the fact that companies have been throwing lots of money at it for a few years with no AlphaFold-sized breakthroughs to show for it. So, adjusting slightly downward, the probability that AI R&D is automated in the next 6 months seems less than 3%.

But even if AI R&D was automated tomorrow, this wouldn’t guarantee that we’d reach a time horizon of 1 year in less than 6 months. It’s more likely that the speed of AI progress would slowly ramp up as people and AIs found better ways to distribute labor and resources between automated and human AI researchers. And training runs sometimes take a while, meaning the breakthrough might take more than 6 months to fully take effect.

Would we notice a massive capabilities increase?

It also seems more likely than not that if any AGI company was on track to create AIs which have dangerously high levels of general capabilities in the next 6 months, they would be able to tell that this is happening, at least at the start. They would see that one year of progress has happened before all five years of progress have happened, at least assuming that there aren’t large discontinuities between checkpoints in training.

If a 1000x increase in time horizon routes through AI R&D automation, then the AGI company would definitely at least notice that AI R&D has been automated. If it doesn’t route through AI R&D automation, it’s likely we’d notice a 10x increase before the 1000x increase. So in any case, the AGI company is likely to have some sense that “something big might be happening” if they’re heading towards a major improvement in capabilities.

However, it is plausible that after some level of capabilities, the AIs would figure out how to subvert oversight mechanisms and make it look like they’re less capable than they actually are. So if capabilities measurements during training are too sparse, or easy to subvert, the researchers might not notice a sudden jump that enables oversight being broadly undermined.

Conclusion

So, judging from the size of breakthroughs needed, and from the sizes of some recent AI breakthroughs, it seems very unlikely (<2%) that AI will reach a 1-year time horizon in the next 6 months. The main pathway I see is a sudden breakthrough in coding, which would lead to automated AI R&D, which would lead to a large number of transformer-sized breakthroughs in quick succession. Accounting for unknown unknowns, I’d increase my probability to around 3%. If this does happen, I think it’s more likely than not than the AGI companies in question would have some awareness that it’s happening, instead of it being a complete overnight surprise.

1. ^

   I’m assuming away the possibility of a catastrophe caused by misuse of AI systems, like bad actors using AIs to create very potent biological weapons. I’ll only consider AI catastrophes caused by autonomous AIs.

2. ^

   By “catastrophe”, I mean an event where 100 million humans die, or something even worse happens.

3. ^

   Although I do feel kind of skeptical of this number. Surely transformers were a bigger deal than that? Without transformers I’d guess we’d be more than 1 year behind, but that’s probably a different operationalization than the one Epoch AI uses.

Discuss

Published on November 11, 2025 12:43 AM GMT

One important property for a style of thinking and argumentation to have is what I call galaxy brain resistance: how difficult is it to abuse that style of thinking to argue for pretty much whatever you want - something that you already decided elsewhere for other reasons? The spirit here is similar to falsifiability in science: if your arguments can justify anything, then your arguments imply nothing.

 

In this post, I will argue that patterns of reasoning that are very low in galaxy brain resistance are a common phenomenon, some with consequences that are mild and others with consequences that are extreme. I will also describe some patterns that are high in galaxy brain resistance, and advocate for their use.

Vitalik talks about styles of arguments that prove too much, pulling examples from the AI, EA, and cryptocurrency communities. As defenses, he recommends having deontological principles that override slick reasoning and avoiding incentives that would distort your reasoning.

His advice to would-be AI safety researchers:

1. Don't work for a company that's making frontier fully-autonomous AI capabilities progress even faster
2. Don't live in the San Francisco Bay Area

Discuss

Published on November 10, 2025 11:59 PM GMT

Why aren't all pencils the same quality, conditional on price?

When I go to a store to stock up on pencils[1], I'm presented with an array of choices and lacking any indicators of quality, I choose any old brand. For some strange reason, the quality of the pencil will be a crapshoot. The lead inside might break easily, or it might not. The mechanism may get stuck after a month, or it may not. The eraser may be a flimsy thing unworthy of the name, or it might match a Staedler. Why?

Is a mechanical pencil not a commodity good, which should be fungible no matter the make? Like fruit, it should be of consistent quality. Wait.

Fruit is not of a consistent quality. Nor are laptops, or shoes or paper, or really any of the goods that pop to my mind. E.g. go buy 10 random types of laptops at the same price point, and I'm sure they'll vary greatly in quality.

So we have a new mystery. Why are commodity goods not the same quality, conditional on price? (Another way to frame this mystery is: why does it pay to look for high quality in so-called commodity goods?)

I'm not talking about a high bar for consistent quality here. If most of the variance was explained by within brand-line variation, then a product would in my mind live up to the name of commodity good. But they don't.

My guesses as to why have the following structure: 
1) Manufacturers mostly aren't optimizing for functional quality, but for other traits. And when you are optimizing for one trait, the other traits you do not optimize take random values. Only the moderate pressures for quality prevent greater variance in quality. This explanation is dual to the next.
2) Consumers care about something other than quality.

2 reminds me of a question Robin Hanson raised: why is there so much diversity in modern products? Why hundreds upon hundreds of different kinds of phones instead of just a few? His guess is that consumers want diversity to signal their own differences from others, which is a high-status behavior now common because we moderns are status mad.

I think this theory has some merits. Certainly, it says why there should be lots of variance amongst brands. And if manufacturers mainly optimize for diversity, then functionality varying greatly makes some sense. At the most extreme ends of fashion, you have clothing that is already torn to shreds.

But I don't think most people are signalling anything with their choice of mechanical pencils. Oh, some do, for example children who want dinosaurs on their pens. But that's a different segment of the market. For all of us who don't attach much care about the designs of their mechanical pencils, shouldn't we get boring, reliable quality?

Still, it's the best answer I've got. Stuff like "consumers just don't care enough to do the research needed for quality" are question begging.

1. ^

   I used to do this, but I've since remedied the problem by researching quality mechanical pencil brands. I've found the Pentel P205 0.5mm to be a worthy weapon with which to challenge Landau and Lifshitz. 

Discuss