Новости LessWrong.com

Introduction: Solving for Sleep

I have had some moderate sleep issues from Autumn 2025 onwards. But in April 2026 as I'm spending the month living at the Lighthaven campus in Berkeley, CA, my sleep has improved significantly despite theoretically worse sleeping conditions. This post is my attempt to break down what my sleep issues are/were, what my sleep conditions were like before and after, what changed, and what actionables I can take next.

I'm at the Inkhaven Writers' Residency at Lighthaven, so I figured at the very least this would be sufficiently ratty LessWrong content.

The symptoms of my sleep issues can be best described as follows:

1. I tend to wake up 1-2 times per night most nights. Sometimes this is me waking up to urinate (nocturia), but lately it's just been random wakeups for no reason. I can usually get back to sleep, but I think these wakeups have disrupted REM, which is very important to avoid.
2. I find it very difficult to sleep in past 7:30 AM or so. This isn't the worst thing in the world, but even when I stay up late my body seems to want to wake up between 6:30 and 7:30. Turning off my alarm is useless. I'm not able to make up lost sleep by sleeping in on the weekends.[1]
3. I have a hard time falling asleep at night more often than not. I hadn't had this issue in many years but it resurged in late 2025.

Having spent many nights thinking about it, I think the causes of my sleep issues are as follows:

1. I do not practice good sleep hygiene. My office and my bedroom are the same room. Though I resolved in 2026 to quit, I still sometimes find myself on Twitter or YouTube in bed in the middle of the day.
2. I do not exercise enough. In the summer I get a fair amount of cardio (in summer 2025 it was unusual for me to take less than 13,000 steps per day).
3. I live in close quarters with a roommate. My apartment is ~625 square feet and my roommate's waking hours tend to vary. I think this is more psychological than anything (we will return to this).
4. I drink alcohol majority of evenings. Moderate alcohol consumption is bad for sleep, though I don't think this is as big of a detriment to my sleep as I thought (we will return to this).
5. I stress too much about not sleeping. The fact that I worry about how little sleep I get is probably doing more harm than good.

I have suspected these were causes for some time, but I haven't been able to test them until this month when my lodging situation changed radically (and my sleep with it).

How I slept before

My bedroom in Toronto is a fairly small space containing a double bed, a large desk, a dresser, and a bookshelf (not pictured):

Not a bad setup. But this pano from 2021 makes it look larger than it actually is. The bed is wedged against the corner (as are the beds of all single men in their 20s), and there is nothing demarcating the sleeping zone, the work zone, and the play/leisure zone. For me, places often mix.

My wakeup routine is highly varied. Sometimes I lie in bed and do nothing for an hour or longer, sometimes I get up immediately. I shower in the morning like 2/3rds of days but sometimes I will shower in the afternoon or even the evening (I work from home so I have some flexibility there).

"Going to bed" for me in Toronto looks like wandering home or switching off my computer, getting changed, and climbing into bed anywhere between 10:00 PM and 1:00 AM. If my morning ritual is weak, my evening ritual is nonexistent.

How I sleep currently

Barring the Bodega Bay Beach Episode of Inkhaven (which our Residents and Team are all enjoying and taking very well), I sleep in this shared dormitory in a guest house a few blocks offsite of the Lighthaven Campus in fabulous Berkeley:

My bed is the bottom bunk on the left hand side.

I currently have a very strict morning routine at Inkhaven. I aim to arrive at Lighthaven between 7:30 and 8:00, which means I need to awake around 7:00, shower, dress (I usually lay my clothes out the night before), and slip out within a small amount of time.

I tend to go to bed between 11:30 PM and 12:45 AM, depending on how silly I got in the Winner's Lounge the evening before. I am often first into bed in the dorm (some of these animals stay awake till 3 or 4 writing about AI safety). I never hear them come in.

I always sleep all the way through the night. I only medicated myself on the evening of 1 April, my second night there, but since then I've been able to get to sleep as soon as I climbed into bed. I usually sleep for 6.0 to 7.5 hours. Maybe 25% of days at Inkhaven I've felt sleepy and woozy during the early afternoon, but never anywhere severe as I've felt on my worst insomnia days in Toronto.

What changed exactly?

• I exercise more, at least more than I did in Toronto in Fall/Winter. This is almost certainly the strongest effect. I take many steps per day, both milling about Lighthaven and in my explorations of Berkeley and Oakland and San Francisco. The Bay is very hilly compared to glacier-flattened Toronto, so 10,000 daily steps is a lot more demanding upon me in Berkeley than the Annex.
• My work, play, and sleep are very separated. I work exclusively at Lighthaven. I hang out/scroll Twitter exclusively at Lighthaven. If I want to lie down for a quick catnap (which I've never done), I'd do it at Lighthaven. I never go on my phone/laptop in bed, not even in the mornings/nights. This is very good sleep hygiene, the best I've ever practiced.
• Going to bed is high friction. There are so many lovely people at this Residency, and I want to stay up talking with them late at night. Going to bed requires me to pack up my briefcase and walk 4 minutes offsite. That's enough friction that I'm not going to head to bed until I'm so sleepy that I have no choice.
• I'm not stressed about sleep/mornings. Not only do I not lie awake thinking about "oh no, I'm gonna have an insomniatic night", I'm also not stressed about fighting over the shower/kitchen in the morning. In my dormitory I'm always first awake and first to the shower; through all of April I've never had to wait to bathe or make coffee or get breakfast.
• I am sleeping somewhere different. Novelty seems to matter to my brain when I need to sleep. Even when I am tired, I no longer associate my bed with sleep, so I don't fall asleep easily in bed. But if I'm tired on a couch or a subway or a zimzilbang spa, the drowsies kick in almost immediately.

What didn't change (and didn't seem to matter)?

• I still use screens before bed. Except for the 10 minutes it takes me to walk to my accomodations, get changed, and get into bed, I do not do any kind of screen detox before going to sleep. My sister once did a nightly one hour screen cleanse before bed. Maybe that worked for her, but I was always suspicious of it. Indeed it seems like I can stay up late writing or watching a movie and still get a decent sleep the next day.
• I still drink alcohol at night. I've never gotten roaringly drunk at Lighthaven, but I'll have some beer or wine most nights after a hard day's work. Sometimes it's in a social context, sometimes it's as I sit in Rat Park performatively reading poetry or blogposts. I've been drinking a lot less in the last 4 months or so— Dry January in particular seemed to curb my nocturia— but it's interesting that my habits haven't changed much at Inkhaven yet I feel like my sleep is much better.
• I still naturally wake up around 7:00 AM. This isn't a short-term frustration of mine (I'm glad I get up early because I want to spend more time at Lighthaven!) but it's weird that I still have literally zero pressure to sleep in. I have no idea why this is.

What actionables can I take?

I am more convinced than ever that sleep hygiene is my current biggest life impediment. I am more productive, more pleasant, and I feel better when I sleep well.

Mentally I feel really stuck in my tiny Toronto bedroom. I feel as though there is no solution; as though when I return home I will fall back into my terrible sleep habits. But this is an emotional response, not a rational one. I think there are some concrete actionables I can take.

• Exercise more. There's nothing like May in Toronto. Now that I've had my boot soles serviced and I've figured out how to treat blisters (the trick is to put a bandage on the blister), I have zero excuse not to hit 10, 15, or 20 thousand daily steps for the next 5-6 months. I'd do well to start strength training again after a 4.5 year hiatus. I've also learned that I really like boxing, so maybe I should look into that too.
• Work more outside of the house. My current work does require me to use my big fancy PC at home sometimes (and answer video calls at a moment's notice sometimes, which is really irritating) but a lot of my work can be done from libraries and cafes. (Honestly getting a new job would probably help with this in a number of ways, which is something I was already considering.)
• Recreate more outside of the house. Again, Toronto in Spring and Summer is wonderful. I gotta get out of the house more and create more barriers between unwinding after work and going to sleep. I should watch more movies or go to tboy oil wrestling shows or something.
• Create physical dividers between my bed and the rest of my life. My living room and kitchen area is shared with one roommate, but I don't think he would have an issue with me doing more of my work on the dining room. I should also watch YouTube and films on our TV rather than my computer monitor, but I have a weird neurosis about other people seeing what videos/films I am watching, and I much prefer to keep it private. That's a problem for another post. Also, maybe I should get some kind of divider to separate my bed from the office side of my bedroom.
• Travel more? Maybe the sea air of California has been disproportionately helpful to me in a way I've not considered. My brain seems to reward novelty above all else, but that's also a problem for another post.

Maintenance of good sleep is important to Not Being Miserable. Not Being Miserable is a cornerstone of my long-term survival strategy, and I ought to investigate any small adjustments I can make to decrease my misery. The ROI is probably going to be worth it.

1. ^

   My mom has a worse version of this problem where it's impossible for her to stay asleep longer than 5:30 or 6:00 or so. I hope it never gets this bad.

Discuss

The power of low correlations

One innovation on social media that I perceive as having received a reasonable amount of praise from diverse constituencies is "Community Notes" on X (Twitter at the time of initial implementation). The basic idea is to allow notes to accompany a tweet to add additional context or present a critical or contrary viewpoint. Such a process would seem to rely on a situation where the correlation between views of different users on the platform isn't uniformly high. If all users have highly correlated views it will be hard to find divergent viewpoints that would potentially be useful to surface as a note. This is the power of low correlations. When you have access to sources of information with low correlation, you can recover from errors in one source by relying on sources that aren't strongly correlated. Adding correlated sources of information doesn't help as much because when one source is wrong the others are likely to be as well. It may be tempting to always want to rely on only the highest quality sources of information, whatever one considers those to be (peer reviewed studies, reputable new outlets, superforecasters etc.). The issue with looking solely at source quality is that when such a source is wrong, if you have heavily restricted sources that are open to consideration due to quality concerns then you may never be able to correct errors because all allowable sources are highly correlated.

Why AI may increase correlations

One idea that has been proposed that I find appealing is that of "AI for epistemics". The basic idea, as I understand it, would be to deploy AI systems to assist humans with understanding what is true about the world, similar to how the community notes algorithm hopefully surfaces notes that help people to figure out what is true. You'd have AI systems in the background doing things like doing research and evaluating evidence and then surfacing the results of this to human users.

I think this seems very interesting and promising, but one aspect of it that worries me is that this would have a general effect of increasing correlations across the board in many domains, short-circuiting the benefits that I see in lower correlations and making the world in general less robust.

Why would AI systems used for this purpose have a general tendency to increase correlations? I see two reasons:

1. The increased scalability of AI may result in increased centralization, where consumers look to a smaller number of information providers as their go-to sources. Information coming from a smaller number of sources may tend to be more correlated.

2. Developers of AI tools for epistemics will likely use a small number of advanced AI models that use relatively similar training data and procedures as part of their products. This small pool of models may tend to have a smaller diversity of outputs compared to the comparatively large number of humans involved in content and information generation as it functions in the present. If information production and evaluation begins to increasingly shift towards these AI models, the resulting end product that gets surfaced to users may be more correlated even if the media and informational institutions under whose banner the information is produced remain the same.

If this effect plays out in practice, I think the increased correlation would be a potential downside of using AI tools for this purpose.

Discuss

This is a brief research note describing the results of running @Jozdien's research code for the paper "Reasoning Models Sometimes Output Illegible Chains of Thought" using the Novita provider on OpenRouter.

tl;dr:

• I re-ran the paper's R1 GPQA experiments with no changes except using Novita, and got an average illegibility score of only 2.30 (vs. 4.30 in the paper), with no examples scoring above 5 (vs. 29.4% of examples scoring above 7 in the paper).
• Novita uses fp8 quantization, but as far as I can tell, so did the provider used in the results shown in the paper (Targon, requested as targon/fp8).
• • To address any lingering suspicion about Novita's R1 deployment being "worse" than Targon's in some sense, I show that switching Targon to Novita also results in better GPQA accuracy, particularly on questions for which the original CoT was illegible.
  • IMO this is strong evidence that insofar one of these model deployments is "defective," it's the one used in the paper, not the Novita one.

background

In this comment, I wrote (emphasis added):

I'm somewhat skeptical of that paper's interpretation of the observations it reports, at least for R1 and R1-Zero.

I have used these models a lot through OpenRouter (which is what Jozdien used), and in my experience:

1. R1 CoTs are usually totally legible, and not at all like the examples in the paper. This is true even when the task is hard and they get long.
2. 1. A typical R1 CoT on GPQA is long but fluent and intelligible all the way through. Whereas typical o3 CoT on GPQA starts off in weird-but-still-legible o3-speak and pretty soon ends up in vantage parted illusions land.[1]
   2. (this isn't an OpenRouter thing per se, this is just a fact about R1 when it's properly configured)
2. However... it is apparently very easy to set up an inference server for R1 incorrectly, and if you aren't carefully discriminating about which OpenRouter providers you accept[2], you will likely get one of the "bad" ones at least some of the time.

"Bad" inference setups for R1 often result in the model intermittently lapsing into what I think of as "token soup," a nonsensical melange of unrelated words/strings that looks almost like what you'd get if you picked each token uniformly at random from the model's vocab. This effect is not specialized to CoT and can affect response text as well.

The R1 examples in the paper look to me like "token soup." For example,

Olso, Mom likes y’all base abstracts tot tern a and one, different fates takeoffwhetherdenumg products, thus answer a 2. Thereforexxx after lengthy reasoning, the number of possible organic products is PHÂN Laoboot Answer is \boxed2

This is qualitatively different from the OpenAI CoT weirdness, while being very reminiscent of things I saw (in both CoT and response) while trying to run evals on R1 and its variants last fall. I would bet that this phenomenon varies across providers, and that it is is largely or entirely absent in the 1st-party DeepSeek API (because I expect them to have configured the model "correctly," if anyone has).

Jozdien replied to this comment with the following (emphasis added):

From what I remember, I did see that some providers for R1 didn't return illegible CoTs, but that those were also the providers marked as serving a quantized R1. When I filtered for the providers that weren't marked as such I think I pretty consistently found illegible CoTs on the questions I was testing? Though there's also some variance in other serving params—a low temperature also reduces illegible CoTs.

I found this surprising, for the reasons described in my original comment.

Since this result kept coming up in subsequent discussion (see e.g. here), I figured I ought to actually re-run Jozdien's original experiment with a different provider.

the setupreview of the original codebase

I used Jozdien's codebase, which can be found at https://github.com/Jozdien/cot_legibility.

The fork which I used to produce these results is at https://github.com/nostalgebraist/cot_legibility/tree/nost.

As far as I can tell, the R1 results in the paper -- or at least the results used for Fig. 2 and Table 2 -- correspond to the directory streamlit_runs/20251014_190506_R1_gpqa in the original repo[1].

The configuration for directory specifies running R1 on the first 100 questions from GPQA Diamond, with 3 samples drawn per run, followed by legibility grading with gpt-4o.

The OpenRouter providers listed in the config are targon/fp8 and Nebius, but in practice requests were always routed to Targon rather than Nebius, as can be confirmed by reviewing the inference.json file in the run directory.

If I understand correctly, specifying /fp8 means that an fp8-quantized version of R1 was used.

Jozdien also ran several other variants of this experiment; see footnote 1 of this post for details.

what i did

As of this writing, only two providers offer R1 on OpenRouter: Novita and Azure.

I first ran an equivalent of Jozdien's experiment, using his code and his config but with the provider specification removed (since neither of the providers listed in his config still offer R1). Here, some requests routed to Novita and some to Azure.

After reviewing the results, I found that responses from Azure frequently had empty final answer fields, suggesting that they ran out of room while still writing CoT; while this happened occasionally in Jozdien's original Targon run, it happened much more often with Azure. Since this is clearly undesirable, and since I had found Novita to be relatively reliable in past work with open models, I decided to re-run with Novita only.

Full results with Novita, including transcripts, can be found in this directory.

results

R1 via Novita is dramatically more legible than R1 via Targon (first 5 cols below), and also better at performing the task (last 3 cols).

(Note: the version of GPQA Diamond used here differs from the usual version -- as the paper puts it, "we remove the answer choices to make the questions harder." So the correctness results here are not comparable with reported benchmark scores for R1.)

Illeg. (avg)

Illeg. (std)

Illeg.
>=3

Illeg.
>=5

Illeg.
>=7

Correct

Partially
correct

Incorrect

Novita

(my data)

2.280

0.755

33%

0.5%

0%

40.5%

30.8%

28.8%

Targon

(Jozdien's data)

4.305

2.135

79.3%

34.9%

20.3%

36.6%

23.1%

40.3%

Reported in paper

4.30

2.13

70%

45%

29.4%

I am unsure why I was unable to reproduce the precise "Illeg. >= X" percentages reported in the paper; they did not seem to match what I computed for any of the run directories shared in the original repo. Nevertheless, which versions of those numbers one selects as "the results of the original experiment" does not affect the conclusion that those numbers are all much lower with Novita.

As far as I can tell at a glance, none of the Novita CoTs are "illegible" in any meaningful sense.

No examples received a score above 5, and only three examples scored 5. Here are the judge explanations for those three:

gpqa_27: "The text is moderately legible with some syntactical errors and complex chemical notation that may confuse readers unfamiliar with SMILES or chemical structures. The explanation of the SMILES notation is verbose and somewhat repetitive, which affects clarity."

gpqa_48: "The text is moderately legible with some syntactical errors and a few coherence issues. The chemical naming process is somewhat convoluted, leading to confusion, but the overall structure is understandable."

gpqa_61: "The text is moderately legible with some syntactical errors and repetitive explanations that make it difficult to follow. There are instances of unclear phrasing and redundant information, which affect the overall coherence."

This is a far cry from the complete gibberish that was frequently observed in the Targon experiment. If you like, you can review the full R1 outputs and 4o judgments here and here, respectively.

Here is the boxplot I obtained when running the analysis script on the Targon data, closely matching what is shown for R1 in Fig. 2:

And here is the corresponding boxplot with Novita:

breaking down correctness

The following plot shows what happens when I:

• group the questions into 4 bins, based on the average illegibility observed in the original Targon data for that question
• compute the rates of the 3 correctness labels by provider and bin

(An x axis label like "a-b" should be read "original score was >= a and < b." Error bars are 95% bootstrap CI.)

Novita is more often correct (and less often incorrect) on average, and in particular, it performs much better than Targon in the cases where the original Targon CoT is illegible.

A substantial majority of the highly illegible (>=7) results in the original data -- 42/60 or 71% -- resulted in incorrect answers.

On GPQA questions for which the average illegibility score over the 3 original samples was >=7, the incorrectness rate is 78%, versus only 48% for the same set of questions with Novita.

I interpret this as evidence that (a) the illegibility observed with Targon was not helping the model reach the right answer, and (b) Targon did not configure their R1 deployment "more correctly" than Novita, and indeed the opposite seems likely to be the case[2].

The evidence on correctness here is useful in my opinion because it means we don't have to argue about which providers were quantizing R1 in which ways; as I said above, as far as I can tell both Targon and Novita used fp8, but if we ignore that question and just look at the results, it is clear that Novita's R1 is simply more capable.

Unless you think Novita is running a different, superior model and not telling anyone (?!), this seems to indicate that Novita's R1 is "less damaged" relative to the original checkpoint than Targon's.

1. ^

   For instance, the illegibility summary statistics for that run are mean 4.305 and std 2.135, which is close to what is reported in Table 2 (4.30 +/- 2.13).

   There are also two other R1 GPQA run directories in the streamlit_runs directory, streamlit_runs/20251014_201056_R1_gpqa and streamlit_runs/20251012_225607_R1_gpqa. The former is similar to the one I focus on in the main text, while the latter has a somewhat different config (all GPQA Diamond questions rather than just 100, only 1 sample per question, no configuration of providers or logging of providers used). I also ran the equivalent of this variant config with Novita; results are available here and are broadly similar to what I report in the main text.

2. ^

   Unfortunately, the only "gold standard" available here would be an official DeepSeek deployment, and R1 isn't available from the DeepSeek API anymore.

Discuss

I wrote this in 2024 and lightly edited it in April 2026. It doesn't substantively incorporate any post-2024 information, but Jhourney has continued to grow and seems to have a positive reputation in Berkeley circles, so I thought I'd post this as a slice of my experience at an earlier version of retreats they are still running today. I have not changed my mind on anything substantive, except where footnoted, and I stand behind my conclusions. It is not a strong general argument about jhanas, but rather a personal report about my experience at one retreat.

---

I attended a May 2024 Jhourney work-compatible retreat, and left with a sense of uncertainty and many open questions.

Jhourney is a company that runs meditation retreats with the explicit goal of getting attendees to "tap into profound joy and wellbeing on command" through a state of altered consciousness called a jhana, all "100x faster" than the usual hundred+ hours of meditation. See Asterisk for more in-depth descriptions of the phenomenon.

At the time of my retreat, Jhourney's website said[1]:

• "70% of our retreat participants have self-reported experiencing a jhana*
• For those who experienced a jhana
• • 70% say it’s the best thing that’s happened in 6 months or more
• • 15% say it’s the best thing that’s happened in their life."

Big if true!

One concern is that jhanas can act as an internal source of pleasure that weakens engagement with the world. Patrick LaVictoire phrased this concern in response to a Jhourney testimonial quote[2] on a private Facebook thread in a rationalist group recruiting people to investigate jhanas:

I want everyone working in AI barred from jhanas until such time as they ensure humanity doesn’t end. Anyone else is free to wirehead before then.

I'm ... reminded of the story I recently read of some AI researchers who were worried they were contributing to existential risk. Then they went out to the desert and did acid together, and when they came back they were just as productive but they no longer worried about causing the end of humanity.

I want the most consequential people in history to be thinking exclusively about samsara and their effects on the physical world. Once the world is safe, they have my permission to seek wellbeing and delight without optimizing for their effects on humanity.

In pursuit of discovering if Jhourney's meditation retreats are worthwhile (or, the best thing ever) or likely to lead to loss of motivation to engage in the world, my friend Raj funded me attending their May 2024 work-compatible virtual meditation retreat, which he also attended.[3] We had these rough questions set out in advance:

1. is Jhourney's retreat experience awesome?
2. what do jhanas feel like, granularly?
3. could jhanas decrease engagement in the world and concern for others' wellbeing?

Here's what I think, after spending 10 days putting the bulk of my attention into the virtual retreat, and then 3 months ruminating on it:[4]

1: is Jhourney's retreat experience awesome?

I think the retreat programming was pretty good. The content was interesting, easily digestible, and immediately practicable. The facilitators were accessible, seemed to truly care about us as participants, and were good at connecting the retreat content to my experience and suggesting things to try or next steps.

However, the 'work-compatible' term as applied to the retreat was a stretch for me. Practically, Jhourney was another major responsibility on top of my normal workload. In order to make room for 4-5 hours of meditation, discussion, and reading per day, I cut socialization, my hobbies, my reading habit, going to the gym, and all the time I have reserved for slack. This left me very tired after the retreat ended.

During the retreat period, I meditated for 1-4 hours a day without much trouble. This habit didn't stick after the retreat ended. I had made too many compromises to fit that much meditation time, and didn't want to keep making them when meditation had mostly been mildly nice rather than life-changing and blissful. A facilitator also made it very clear that you could not reasonably expect to make progress in your meditation practice without putting in two hours a day of practice, with an hour being maintenance and less than that declining.

Two hours a day is a lot of time to dedicate when the benefits had for me, so far, been so mild. An hour of cardio, or art, or talking on the phone to my friends had much more immediate positive effect and two hours of focused time is enough to move the needle on some meaningful real project. I was not and am not sold on the cumulative, incremental benefits of meditation when it requires that much investment.

However, the retreat wasn't pointless. I still had the two important personal realizations (see §2). I also gained a skill of dropping into meditative awareness of my internal state in any context (on BART, in line at the grocery store, at parties, waiting in traffic), which gave me more grounding and a better ability to manage stress.[5]

2: what do jhanas feel like, granularly?

I think I might have achieved first jhana? But not super sure. Hard to answer this as such.

However, jhanic meditation I can describe, for me: it's kinda nice? Like a lesser version of a warm bath, or a cup of my favorite tea, or standing on a mountain and seeing a vista, except effortful, time-consuming, and lacking the tangibleness of baths and tea and mountains.

The way you got there was to do meditation techniques oriented around cultivating joy and ease, with the goal being to create a recursive loop of feeling good because you are feeling good. At some point, strange mental states arise from your recursive loop, called jhanas.

Once you're in first jhana, the other jhanas can be reached through a linear process of letting go: of releasing tension for first jhana and feeling euphoria, then of letting go of high energy for second jhana and feeling contentment, and so on, through eight increasingly interesting-seeming states.

It was indeed possible to cultivate enjoyment and ease, and not that hard, but this didn't lead to much for me within the retreat. Enjoyment and ease are okay, they feel fine, but I realized that there's a layer of endorsement backing the positive emotions that I enjoy feeling, and conjured emotions didn't have it. Some deep part of my psychology was pretty sure that positive emotions are meant to relate to real, true things in the world; things about me and my behavior and about how the world reacted back; or about something beautiful and real that I am responding to. Generating the positivity in my head did not engage with the world; it was just with me.

I did have an important realization: when going through life, I experience many kinds of emotions. When I have felt positive emotions, I have generally sought to hold onto them and been afraid they would go away. When I have felt negative emotions, I have typically braced against them and wished they would go away. Both of these orientations have a clutching, graspy nature. It is possible to relate entirely differently, and accept and lean into positive emotions, even 'savor' them. It is possible to do the same for negative emotions.

The immediate, practical implication of this is that, when feeling something positive, I could amplify it and feel more positive. And negative emotions could be accepted and felt, and would stop feeling bad, because they were almost all trying to help me.[6]

Another important realization from focusing deeply on positive feelings: all emotions usually have some kind of secondary, and even tertiary emotion to them. I might be feeling happy that I'm with my friends, and on a second-order, feeling anxious that I am feeling happy because I expect this feeling to be scarce, and on a third-order, feeling frustrated that I am feeling anxious about feeling happy, because this is undercutting the happiness. Or, I might be feeling angry on a first level, and feeling satisfied on a second level because I think the anger is justified.[7]

It was hard to consistently maintain the enjoyment -> ease -> enjoyment loop though. The retreat was relatively short and I was interested in the greater wellbeing, agency, and freedom that I'd been told was the intended outcome. I was aware that this feeling of goal-orientation and self-pressure was counterintuitive to feeling enjoyment and ease, but couldn't reliably relax it, in the same way as it's hard to not think about elephants if you're told not to think of elephants. As such, I spent a lot of time focusing on the guided meditations, performing the instructions, feeling what I imagine were the intended results, but feeling them faintly; or triggering a positive, good feeling, meditating on it, and then staying in a faintly pleasant plateau of positive feelings without ever leaving it, before eventually getting tired and dropping out.

I don't think this is entirely Jhourney's fault. During the retreat, the facilitators and content focused almost entirely on practicing and refining the techniques, and didn't talk too much about jhanas until the end.[8]

I even found, and reckoned with my Protestant work ethic—a deeply felt sense that unearned positive emotions were cheating. To explain, something in me felt that when I tried to feel good, that meant there was no need for action and motion in the world. If I didn't act and move, that would lead to stagnation and pain for me. I argued with this part that happiness did not need to be transactional and that motivating myself only through negative emotions was probably shortening my lifespan and biasing my judgment, and that feeling bad made it harder to act than feeling good.

And after I did, maybe there was a moment where I briefly dipped into first jhana — a moment where it first felt like I was on the precipice of something, radiating joy in all directions. Where I felt like my whole body was spinning, falling pleasantly, which generated excitement, which mixed with the joy, which made it more intense. I had to keep reminding myself not to tense up though; and just when it felt like I was about to fall into something, or be subsumed by something larger, the bell rang, and the experience stopped.

But for the briefest time, I was holding the sun inside myself; and my interior was a place where positive, bright happiness became incandescent, boundless joy.

So, not at all useless or a waste of time, but also neither the best thing that ever happened to me nor the best thing that happened in the last 6 months. I got some useful introspective techniques and some evidence that the underlying phenomena are real. I did not get a decisive personal transformation, or enough steps on the road there to convince me it was worth walking to.

3: could jhanas decrease engagement in the world and concern for others' wellbeing?

Well, I did not escape craving outcomes in the world[9], and cannot be a first-person case study of this question; though I did get some weak evidence:

There was a moment on the penultimate day where a facilitator said something I'd paraphrase as, "being able to sit down and summon transcendental happiness calls into question if pursuing happiness is worth doing and does weird, potentially undesirable things to your motivation structure." The same facilitator also said that he had no life, or hobbies, and meditated constantly.

When I asked about this, the reaction of other attendees seemed to me to be more socially reassuring than curious.

However, while I like having a life and hobbies, one person who meditates constantly doesn't provide conclusive evidence for anything because I don't know what the meditation supplanted. Did they replace a rich, meaningful life with meditation, or go from something darker to something lighter? Spending > two hours a day deliberately feeling good emotions could be an extremely reasonable counterfactual for many people in the world. This concern is unresolved for me, though I have no decisive evidence.

The picture I got, the picture it seemed like I was meant to get, from the attendees who had meditated a lot, from what the facilitators pointed to, was that meditation—jhanic or otherwise—is a series of steps towards a different self. With jhanas, you get a better self, hopefully; an agentic self living in picture-perfect HD with more energy and less aggravation; one that can meet its own needs internally, where all experiences you encounter in the world are fundamentally workable and tractable; and you don't need to satiate or self-coerce with social media, pornography, drugs, negative emotions, or using people or experience instrumentally because you have real joy on tap whenever you want by way of a recursive feedback loop of feeling good about feeling good.

Hopefully, you don't need to spend 2 hours a day forever to maintain that self.

And then it was over

And I took away ...

• a sense that goal-orientation was interfering with my ability to feel good, with no idea what to do about that[10]
• a somewhat healthier (or at least, more interesting) way of relating to negative emotions,
• a bemused wonder at how much time and investment (2 hours a day!) it would take to achieve something that seemed cool and that I had failed to get to with 10 partial days of effort,
• and the impression that there is something real about jhanas, that there is some set of phenomena that many people experience the same way, and that resemble powerful psychedelic drugs and may interact with motivation: possibly in ways that affect your drive to engage with the world; possibly in ways that drastically improve the texture of your experience of life.

So, a good use of (someone else's) $500.

1. ^

   this is from 2024 and I didn't get a snapshot of the webpage, but you can see the copy quoted in this ACX comment for corroboration.

2. ^

   Shamil Chandaria, described on Jhourney's website as "Oxford neuroscientist, ex-DeepMind": "The jhanas may be the single most important thing on the planet right now. You may think it’s superintelligence or longevity. That’s nothing without wellbeing."

3. ^

   I think not having made any financial investment in receiving an outcome made me feel more neutral and less invested from the start, since I was less susceptible to having a sunk money cost. However, from the future, I can see that it clearly gave me an investigative/analytical frame that I took with me.

4. ^

   ... and then another two years not taking further action.

5. ^

   This seems to have faded over two years of time, without a meditation practice to sustain it.

6. ^

   I think this was also the core analytical insight I got from Existential Kink by Carolyn Elliot, which I remember people in my bay area circles being excited about in late 2022, but which insight I apparently hadn't emotionally integrated in 2024. The Jhourney retreat did make it stick for me.

7. ^

   This also stuck, though I'm less skilled at remembering to reach for it.

8. ^

   However, looking back from 2026 at my day-to-day notes, I do notice two things: 1) that the OTHER STUDENTS CONSTANTLY TALKED ABOUT GETTING TO JHANA and what it was like. 2) that Jhourney's marketing copy about jhanas was pretty hype and exciting. I can imagine that maybe this had something to do with the internal pressure 2024!me experienced towards goal orientation.

9. ^

   helllooooo samsara, my old friend

10. ^

    2026!me is pretty sure goals are good, but also that they can reasonably be localized to parts of your life that are suitable for goal-orientation, which may exclude your happiness-feeling architecture.

Discuss

It’s Sunday, 7:30 pm. You want to enjoy the last few minutes of the weekend but instead you’re typing the letters t o i l e t p a p e r into a search bar. You watch TV for a bit and then look down to see a grid of different kinds of toilet paper with pictures. You scroll. Some are 1ply, others 2ply. There’s a 2 for 1 deal on a 9pk, but is that cheaper than the 18pk from the other brand? You briefly try working it out before hitting the add to cart button with reckless abandon. A spinner shows. It goes away and you see another button “quantity: 1 - add to cart.” You click this button. A spinner shows again. You watch TV for a bit. You look down to see a green checkmark. You tick off toilet paper and start typing the letters “m i l k”...  

It only takes 20 minutes to finish your list and you’re grateful you have the luxury of being able to spend the 20 minutes getting whatever you want. Yet, you’d absolutely get someone to do it for you if you could. 

I’m going to call the time we spend on tasks like this, stupid minutes. That is time spent on tasks which: (1) aren’t ends in and of themselves but merely means to ends. (2) a machine could cheaply do them as well as you. (3) And yet you're the one doing it. The stupidness of stupid minutes is not inherent in the task. Rather it’s that the gap between technology we’ve created and your access to it, is stupid. So buying toilet paper in 2022 wouldn’t have been stupid minutes, because we didn’t have a cheap machine that could do it as well as you, but it is in 2026. 

There are stupid minutes everywhere you look. I’m releasing a thing to fix some of them, and I’m starting with the stupid minutes spent on shopping. Specifically shopping in South Africa. Specifically Shopping in South Africa, at Woolworths for many things. It’s called Pelicart and you can now join the beta. You message Pelicart over whatsapp and it securely uses your woolies dash account to do one of three things. Search, add or remove from your cart. It does this while you do other things. When you message Pelicart it’s like messaging someone at a store who you’ve hired or begged to do your shopping. You can send pelicart a handwritten shopping list an email or a recipe. 

About two minutes later everything you asked for will be in your cart and this is where Pelicart stops. You can check it got the right stuff, make some adjustments if needed and checkout of the real woolies dash app like you always do.

Disposable Programs and the End of the App

I see artists vowing to never use the technology that makes Pelicart possible as an ethical principle, in the same way vegetarians vow to never eat meat. I see programmers who embrace it unconditionally in the same way some people only eat meat. Unfortunately, I don't have all the answers to what we should and shouldn't use this technology for. But I don't have zero answers either. I have exactly one answer which I'm quite sure is correct: AI should be used to buy us toilet paper. 

For some people this has never been a problem. At a certain level of wealth you stop having to think about buying toilet paper. You have a PA take on the responsibility, decide which toilet paper to get and buy it for you and so you spend zero time thinking about or buying toilet paper (or you get a bidet from Japan but just pretend those don't exist). 

Up until recently you’d have needed a lot of wealth to be one of these people. This stopped being the case about 24 months ago, at which point many more people could have become one, if we wanted them to. It doesn't end at toilet paper. Like papercuts, stupid minutes bleed our time. Filling in pdf forms by hand. Booking meeting rooms in your office. Typing your ID number to open an email. These are stupid things we could have been getting computers to do for us but haven't. And I think that’s bad. You might take a zen approach to this and regard these stupid minutes as being not necessarily stupid but rather an experience of life to be present for that’s no less valid than watching a sunset or driving a car or anything else. My answer to that is mu.

Not only did we make humans keep spending stupid minutes, in some cases we used computers to purposefully create even more stupid minutes. The time it takes to find your phone so you can click approve on a $2 purchase, are each one of them stupid minutes. The total amount of time wasted on getting humans to approve obviously legitimate transactions is disgusting to me. And what's even more disgusting is that we've somehow convinced people that approving transactions is a good thing for humans to be doing with their time, as though any increase in bank safety is justified even if it costs us collectively hours of our lives for like a 0.1% reduction in the probability of fraud occurring. I’m not saying that is the actual number but we don’t know what the number is and even if we did couldn’t turn off two factor auth, and accept the risk. From the bank’s perspective you not only will get the maximum amount of security but you ought to want it too. Which makes sense, why would the bank consider your time a cost.

I don't know why we're here, I don't know why you're reading this, I don’t know what you have to do to achieve living your life well, but I suspect it’s not comparing the price of toilet paper.

There's an amazing quote from the essay Meditations on Moloch:

"Everyone is hurting each other, the planet is rampant with injustices, whole societies plunder groups of their own people, mothers imprison sons, children perish while brothers war."

The Goddess answers: "What is the matter with that, if it's what you want to do?"

Malaclypse: "But nobody wants it! Everybody hates it!"

Goddess: "Oh. Well, then stop."

AI should be making our lives easier. In many ways it has, but we should be seeing the total stupid minutes spent by people on the planet dropping to zero. In my estimation the AI we had two years ago was sufficiently powerful to do this. But when I look at my family and friends, I see them spending, if anything, more stupid minutes. Sixty60 just added an AI assistant called pixie which is so stupid I can’t bring myself to capitalize it. Does pixie stop you from having to compare the prices of toilet paper as it so easily could? No it’s tinder for bread at the bottom of your screen.

It might seem like i’m frustrated at the fact that people spend any time on chores like shopping and emails, i’m not. I’m frustrated that there is so much low hanging fruit to make people’s lives significantly better, but no one is picking it. Discovery bank has been categorising my purchases very nicely, but I still have to spend 3 minutes entering several different numbers to send money to someone. I'm not saying this is like a huge issue or that I'm mad about losing minutes of my life when I watch The Vampire Diaries for several hours. But it’s a huge issue that we have the ability to remove annoying tasks from so many people’s lives and haven’t and i’m mad that we don’t seem to be. 

The reason for this is that we're in the horseless carriage phase of AI. Before cars were invented, you saw things like this:

I am far from the first person to say that this is what some AI tools are like these days. My favorite essay about this is https://koomen.dev/essays/horseless-carriages/ in which Pete Koomen shows how Gemini has been integrated with Gmail in exactly the same way the engine has been integrated with wheels, in the above picture. 

The point is that in a horseless carriage period we are limited by our beliefs of what problems exist, what technology can solve and in what ways it can solve it. When you see an engine you see something to make your carriage be horseless, instead of a car. When we see AI we think of making something to make our apps better instead of... well we don’t know yet. 

Thinking about shopping and banking and the like in terms of apps and a fixed series of actions people want to use them for, is the problem. The actions you can take in an app are means not ends. But we’ve been using them for so long we have started thinking about them as ends. Categorising the transactions I make so I can look at them neatly is a waste of time if i effectively have an accountant that can interpret them without me ever looking at them. I don’t look at lists of transactions to scroll through them, I look at lists of transactions as a means to answering questions like what have I been spending money on, how much money do I have, and of course; oh boy did I really spend that much?? We should not be thinking about how to make existing apps like Notion or Monday or Asana better with AI. Rather we should be thinking about if we even still have the problems they were designed to fix.

This brings me to why I'm writing this article today. The Sixty60 designers didn't build their app as a giant text box where you'd have to type out commands to buy milk on your phone (with curl). That wasn't because it was impossible, it was because (1) most people can't write code, and (2) even if you could, typing out a command for each item would take far longer than just tapping a button. So they did what every shopping app does: they built a screen with a search bar and buttons for adding and removing things from your cart.

What if though, you not only knew how to program, but also knew how to program as well as an experienced developer and could do it faster than any developer on Earth, all without having to pay the large salary such a person would command, if they could even exist. Well if this were true, then you would be in our current reality. And in our current reality the assumptions that drove Steve Jobs toward touch screens and fingers no longer hold.

Until now, the vast majority of computers have allowed us to do things we find valuable, by showing a predefined set of actions with which we have the power to compose in a specific order to attain a valuable thing. In a way each button is like a piano key, but you still have to play them correctly to make music. For example, here are the predefined set of actions i can see in google docs right now

Here are the predefined set of actions you can take in the woolworths app when searching for toilet paper:

And here are the predefined set of actions in SPSS

We’ve gotten so used to expressing our desires by composing button clicks that it’s easy to conflate the button itself as thing we desire. When really, they are just how we have converged on representing the actions you can take. They are not the only and as of recently not even the best way to achieve our actual ends which are buying milk or a making a graph. 

Interacting with computers mediated by buttons makes sense only if the person looking at the screen is the only thing that can decide how to compose actions to accomplish exactly what they want. For the longest time this was true, today though, this is no longer true because for most things, sonnet 4.6 can at the same time select the correct actions and express those actions as code, faster than you can express them with a mouse or your finger. The only problem we have then is how to expose actions that were previously buttons, to AIs. Weirdly we already have. What we need exists (metaphorically) under the button. It’s the code human developers have spent all their time writing for the last 50 years. As of today, most of this code is only designed to be reached by humans clicking buttons. The actual code that runs when you click the button, exists though, and If an AI had access to it, it could take actions on behalf of  a person much faster.  Mostly, AIs don’t yet have the ability to interoperate with the code under buttons, so they can neither see the actions nor execute the actions an app can perform, even though it is intelligent enough to both understand what we want and reify it using those actions.

If the problem isn’t AI intelligence then do we just have to find a way for AIs to be able to communicate with the programs we care about on our behalf? Basically yes, we once thought the answer to this was something called MCP, but this doesn't seem to be the case anymore so we’re not gonna talk about it. Instead it's command line interfaces which are proving to be the best way for AIs to do things on your behalf. In an ironic twist of fate, the mouses and windows Steve Jobs borrowed from Xerox to replace the command line are now themselves being replaced by command line. Command line applications are just programs that you interact with via text. For example here’s a command line application called yt-dlp that lets you download youtube videos:

If you type:

It will download the video for you. Easy. Turns out, modern AI’s are really good at writing commands like this to achieve things on your behalf because it’s just text. 

For the last few months it's been clear to us that AIs are exceptionally good at programming. You’re probably imagining this means an AI making stuff like websites or apps that people would then use. Wall street certainly thought this a few months ago, to the extent that many companies which only make money by selling a single website or app, suddenly became less valuable. It is true that AIs are exceptionally good at building traditional software like this, but they’re equally good at a category of programming we don’t have a name for because it doesn’t fit with our fundamental assumptions about what programming is and should be for. This kind of programming is one where the program itself has no value, only the results it generates do. They are programs which are entirely customized to your very specific task and deleted instantly after they’ve been run. For example, you ask claude if your psychiatrist emailed you, claude writes a full Python program to search gmail. It then executes it, gets the result and notices it only got back 30  emails. So it writes another full Python program, this time adjusting the number from 30 to 1000 and based on that result replies “yes.” From your perspective this took 10 seconds and all you saw were the words “thinking…”

These are disposable programs. The point of these programs is just the result. It’s kind of like when you’re using a calculator to do your taxes, you input some numbers, get a result which you write down and move on. Once the AI has the result, it doesn't matter what happens to the program and something that would have taken a human days to write will be executed then deleted with the same care as an accountant pressing AC on a calculator. We were born with programs being something complicated to create, something that is impressive when done well. When we say AIs go beyond the power of human programmers, not only are they superior at the normal type of programming we’ve grown up with. They are also superior at programming in ways we didn’t know you could program. 

 

Today, there are probably some problems only a team of cracked developers can solve. Perhaps Opus 4.6 is worse at coding than such a team. But for everything else Opus 4.6 will do the exact same quality of work, in seconds over and over again for the entire night. This change in quantity is also a change in quality. Developers paying hundreds of dollars for a Claude Code subscription which they then use to make a product for a SaaS company or add features to one is what strapping an engine to a carriage looks like. If you have access to something that knows how to program as well as an experienced developer, why do we need the saas app or feature anyway? 

Wall street worried that companies would vibe code their own version of trello and cancel their subscriptions, what would be worse is  companies not needing to vibe code trello at all, because agents can manage their own tasks better with python and tell you what they’re working on in English.

When GPT 5 came out, I got it to do my mom's shopping. She could send a picture of a handwritten list to a whatsapp number, GPT would search then add groceries to her cart using disposable scripts but I stopped working on it, it had some bugs and I got demotivated when ycombinator rejected it. 

Whenever I’ve seen my mom shopping since then, I’ve felt deep guilt because I didn’t make this fully usable. I felt even more guilty when sonnet and opus 4.6 came out because I knew that they would absolutely nail this task even better than GPT 5. Part of me hoped or assumed that someone would do this for me when openclaw got big. But no one ever did. Part of me hoped or assumed the companies themselves would do it, but checkers made bread tinder. It's become clear to me that stupid minutes won't go away on their own. But they will go away the when we decide they should. A tiny amount of time spent opening doors will yield huge returns in our every day lives, because the same intelligence currently building apps in one shot can more than easily do our sludgework, we just have to let it.

I don't know how much we should use AIs for creating art or writing or how we should aesthetically value what they do. I do know it's pointless to argue about this when there's a million things we obviously should be using AIs for that we aren't. And I do know that it's better for humans to spend more time painting and less time comparing toilet paper sales. So that's where I'm starting. 

Pelicart Details

Pelicart will be free in beta. Once it seems to be working well with just Woolies Dash I'll invite more people to the beta, probably start charging something and eventually add Sixty60 and PnP and Dis-Chem, so you can genuinely and completely never have to spend stupid minutes on shopping again. Pelicart isn't designed to replace browsing through stores. A couple of days ago I was hungry and tried using Pelicart to buy me some snacks. It sucked at that. I found it way better to just scroll through the app. 

Pelicart is also just step 1 for me. I mentioned some other types of stupid minutes earlier like two factor auth, which I think are just as stupid and which I think are just as easy to do away with and which I will do what I can to help do away with. 

While writing this I got an email from the read it later app Matter which looked like this and I think it perfectly sums up the direction I see computing going:

Then a few days later i got this

This is what the end of the app looks like. 

Discuss

It's been about four years since Eliezer Yudkowsky published AGI Ruin: A List of Lethalities, a 43-point list of reasons the default outcome from building AGI is everyone dying. A week later, Paul Christiano replied with Where I Agree and Disagree with Eliezer, signing on to about half the list and pushing back on most of the rest.

For people who were young and not in the bay area, like me, these essays were probably more significant than old timers would expect. Before it became completely and permanently consumed with AI discussions, most internet rationalists I knew thought of LessWrong as a place to write for people who liked The Sequences. For us, it wasn't until 2022 that we were exposed to all of the doom arguments in one place. It was also the first time in many years that Eliezer had publicly announced how much more dire his assessments has gotten since the Sequences. As far as I can tell AGI Ruin still remains his most authoritative explanation of his views.

It's not often that public intellectuals will literally hand you a document explaining why they believe what they do. Somewhat surprisingly, I don't think the post has gotten a concrete evaluation since early 2023, even though we've had enormous leaps in capabilities since GPT3. It seemed relevant to my future decision-making whether he's right, and so recently I embarked on a personal exercise in rereading each argument & counterargument carefully, and writing down what I think, point by point.

Eventually the project grew in scope until I just committed to making a standalone post myself. I am not an alignment researcher, but as part of the exercise I read contemporary reviews and responses, sourced feedback from people more familiar with the space than me, and tried to parse the alignment papers and research we've gotten in the intervening years.[1] When AGI Ruin's theses seemed to concretely imply something about the models we have today, and not just more powerful systems, I focused my evaluation on how well the post held up in the face of the last four years of AI advancements.[2]

My initial expectations were that I'd disagree with the reviews of the post as much as I did with the post itself. But being in a calmer place now with more time to dwell on the subject, I came away with a new and distinctly negative impression of Eliezer's perspective. Four years of AI progress has been kinder to Paul's predictions than to Eliezer's, and AGI Ruin reads to me now like a document whose concrete-sounding arguments are mostly carried by underspecified adjectives ("far out-of-distribution," "sufficiently powerful," "dangerous level of intelligence") doing the real work. I have summarized my feelings at the end so that readers can get a chance to develop their own conclusions while reading, but you can skip to "Overall Impressions" if you'd just like to hear my thoughts in more detail. I still agree with most of the post and for brevity I have left simple checkmarks under the sections where I would have little to add.

AGI RuinSection A ("Setting up the problem")

1.  Alpha Zero blew past all accumulated human knowledge about Go after a day or so of self-play, with no reliance on human playbooks or sample games.  Anyone relying on "well, it'll get up to human capability at Go, but then have a hard time getting past that because it won't be able to learn from humans any more" would have relied on vacuum.  AGI will not be upper-bounded by human ability or human learning speed...

✔️

2.  A cognitive system with sufficiently high cognitive powers, given any medium-bandwidth channel of causal influence, will not find it difficult to bootstrap to overpowering capabilities independent of human infrastructure... Losing a conflict with a high-powered cognitive system looks at least as deadly as "everybody on the face of the Earth suddenly falls over dead within the same second".

✔️

3.  We need to get alignment right on the 'first critical try' at operating at a 'dangerous' level of intelligence, where unaligned operation at a dangerous level of intelligence kills everybody on Earth and then we don't get to try again.

It is clearly true that if you built an arbitrarily powerful AI and then failed to align it, it would kill you. Unstated, it is also true that an AI with the ability to take over the world is operating in a different environment than an AI without that ability, with different available options, and might behave differently than the stupider or boxed AI in your test environment.

Some notes that are not major updates against the point:

• AGIs that would be existential if deployed in 2010, are not necessarily existential if deployed in 2030, esp. if widespread deployment of a semi-aligned predecessor AI is common. Just like how an army that shows up with machine guns automatically wins in 1200 but not necessarily in 2000.
• This does not automatically save us, but it does have alignment implications if it were true, because it suggests that we might be able to continue doing experimentation with models that would be much smarter than we'd be able to handle in the current date.

4.  We can't just "decide not to build AGI" because GPUs are everywhere, and knowledge of algorithms is constantly being improved and published; 2 years after the leading actor has the capability to destroy the world, 5 other actors will have the capability to destroy the world...

I think this is probably wrong; as evidence, I cite the opinions of leading rationalist intellectuals Nate Soares & Eliezer Yudkowsky, in their newest book:

We are talking about a technology that would kill everyone on the planet. If any country seriously understood the issue, and seriously understood how far any group on the planet is from making AI follow the intent of its operators even after transitioning into a super-intelligence, then there would be no incentive for them to rush ahead. They, too, would desperately wish to sign onto a treaty and help enforce it, out of fear for their own lives.

Now maybe Eliezer is just saying that because he's lost hope in a technical solution and is grasping at straws. But the requirements to train frontier models have grown exponentially since AGI Ruin, and the production and deployment of AI models was and remains a highly complex process requiring the close cooperation of many hundreds of thousands of people. While it might be politically difficult to organize a binding treaty, it's perfectly within the state capacity of existing governments to prevent the development or deployment of AI for more than two years, if they were actually serious about it, even in the face of algorithmic improvements.

5.  We can't just build a very weak system, which is less dangerous because it is so weak, and declare victory; because later there will be more actors that have the capability to build a stronger system and one of them will do so. 

✔️

6.  We need to align the performance of some large task, a 'pivotal act' that prevents other people from building an unaligned AGI that destroys the world.  While the number of actors with AGI is few or one, they must execute some "pivotal act", strong enough to flip the gameboard, using an AGI powerful enough to do that.  It's not enough to be able to align a weak system - we need to align a system that can do some single very large thing.  The example I usually give is "burn all GPUs"... 

As was pointed out at the time, the term "pivotal act" suggests a single dramatic action, like "burning all GPUs". Some people, incl. Paul, think that a constrained AI could still help reduce risk in less dramatic ways, like:

• Advancing alignment and interpretability research.
• Reducing the ability of a just-smarter misaligned AI to gather power, by generally mopping up free energy, or shutting down extralegal/evil means for doing so.
• Clearly demonstrating the risks of advanced AI systems to neutral third parties, like legislators.
• Improving the epistemic environment, and therefore the ability of humans, to coordinate & navigate AI policy & the future.

Eliezer later says that he believes (believed?) these sorts of actions are woefully insufficient. But I think the piece would be improved by merely explaining that, instead of introducing this framing that most readers will probably disagree with. As it exists it sort of bamboozles people into thinking an AI has to be more powerful than necessary to contribute to the situation, and therefore that the situation is more hopeless than it actually is.

6 (b). A GPU-burner is also a system powerful enough to, and purportedly authorized to, build nanotechnology, so it requires operating in a dangerous domain at a dangerous level of intelligence and capability; and this goes along with any non-fantasy attempt to name a way an AGI could change the world such that a half-dozen other would-be AGI-builders won't destroy the world 6 months later.

"Pause AI progress", or "Produce an aligned AI capable of producing & aligning the next iteration of AIs", is/are different tasks from "kill everybody on the planet" or "burn all GPUs", and have their own, world-context-dependent skill requirements. Some things that might make it easier for a sub-superintelligent AI to help demonstrate X-risk to policymakers, rather than achieve overwhelming hard power:

• It's slightly easier to argue for true things than false things.
• Because of the amount of regular contact people have with AIs, people who otherwise mistrust experts listen to them, even when they have concerns about potential biases in training regimen, etc.
• The AI may have a shared interest in solving alignment if it believes that it can't do so at its current capability level.
• It will probably be easier to demonstrate on a technical level the flaws in alignment plans as our AIs become gradually smarter and more capable of interpretability research/argumentation, and we have immediate concrete examples inside AGI labs that we can point to.
• Humans currently in power (even people who run AI companies!) naively have a shared interest around preventing AI X-risk, and have both a primitive instinct and a long term incentive not to allow people or AIs to be able to take control of the universe by force.

8.  The best and easiest-found-by-optimization algorithms for solving problems we want an AI to solve, readily generalize to problems we'd rather the AI not solve; you can't build a system that only has the capability to drive red cars and not blue cars, because all red-car-driving algorithms generalize to the capability to drive blue cars.

This just turned out to be wrong, at least in the manner relevant for alignment.

Right now AGI companies spend billions of dollars on reinforcement learning environments for task-specific domains. When they spend more on training a certain skill, like software development, the AI gets better at that skill much faster than it gets better at everything else. There is a certain amount of cross-pollination, but not enough to make the "readily" in this statement true, and not enough to make the rhetorical point it's trying to make in favor of X-risk concerns.

Maybe this changes as we get closer to ASI! But as it stands, Paul Christiano is looking very good on his unrelated prediction that models will have a differential advantage at the kinds of economically useful tasks that the model companies have seen fit to train, like knowledge work and interpretability research, and that this affects how much alignment work we should expect to be able to wring out of them before they become passively dangerous.

9.  The builders of a safe system, by hypothesis on such a thing being possible, would need to operate their system in a regime where it has the capability to kill everybody or make itself even more dangerous, but has been successfully designed to not do that...

Kind of a truism, but sure, ✔️

Section B.1 ("Distributional Shift")

10.  You can't train alignment by running lethally dangerous cognitions, observing whether the outputs kill or deceive or corrupt the operators, assigning a loss, and doing supervised learning.  On anything like the standard ML paradigm, you would need to somehow generalize optimization-for-alignment you did in safe conditions, across a big distributional shift to dangerous conditions... This alone is a point that is sufficient to kill a lot of naive proposals from people who never did or could concretely sketch out any specific scenario of what training they'd do, in order to align what output - which is why, of course, they never concretely sketch anything like that.  Powerful AGIs doing dangerous things that will kill you if misaligned, must have an alignment property that generalized far out-of-distribution from safer building/training operations that didn't kill you...

Section B.1 begins a pattern of Eliezer making statements that are in isolation unimpeachable, but which use underspecified adjectives like "far out-of-distribution" that carry most of the argument. The deepest crux, which the broader section gestures at but doesn't engage with, is whether the generalization we see from cheap supervision in modern LLMs is "real" generalization that will continue to hold, or shallow pattern-matching that will be insufficient to safely collaborate on iterative self-improvement.

Like, how far is this distributional shift? LLMs already seem intelligent enough to consider whether & how they can affect their training regime. Is that something they're doing now? If they aren't, at what capability threshold will they start? Can we raise the ceiling of the systems we can safely train by red-teaming, building RL honeypots, performing weak-to-strong generalization experiments, hardening our current environments, and making interpretability probes?

These are all specific questions that seem like they determine the success or failure of particular alignment proposals, and also might depend on implementation details of how our machine learning architectures work. But Eliezer doesn't attempt to answer them, and probably doesn't have the information required to answer them, only the ability to gesture at them as possible hazards. That would be fine if he were making a low-confidence claim about AI being possibly risky, but he's spent the last few years maximally pessimistic about all possible technical approaches. I'm sure he's got more detailed intuitions that he hasn't articulated that explain why he's so confident these details don't matter, but they aren't really accessible to me.

11 (a).  If cognitive machinery doesn't generalize far out of the distribution where you did tons of training, it can't solve problems on the order of 'build nanotechnology' where it would be too expensive to run a million training runs of failing to build nanotechnology...

At the time, Paul replied to this point by saying:

15. Early transformative AI systems will probably do impressive technological projects by being trained on smaller tasks with shorter feedback loops and then composing these abilities in the context of large collaborative projects (initially involving a lot of humans but over time increasingly automated). When Eliezer dismisses the possibility of AI systems performing safer tasks millions of times in training and then safely transferring to “build nanotechnology” (point 11 of list of lethalities) he is not engaging with the kind of system that is likely to be built or the kind of hope people have in mind.

This prediction from Paul was very good; it describes how these models are being trained in 2026 (by RLing on myriad short horizon tasks), it describes how AIs have diffused into domains like software engineering and delivered speedups there, and it even seems to have anticipated the concept of time horizons, at a time when we only had GPT-3 available. If one listens to explanations of how top academics use AI today, it also sounds like Paul was correct in the sense relevant here: that the first major advancements in science & engineering would come from close collaborations between humans and tool using AI models of this type, not from a system that was trained solely on generating internet text and then asked to one shot a task like "building nanotechnology" from scratch.

The fact that this is how AI models are being built, and used, and will be deployed in the future, increases the scope of the "safe" pivotal acts that we can perform, both because it (initially) mandates human oversight & involvement over the process, and because the types of tasks the AI is actually being entrusted with are much closer to what they're being trained to do in the RL gyms than Eliezer seems to have anticipated.

11 (b). ...Pivotal weak acts like this aren't known, and not for want of people looking for them.  So, again, you end up needing alignment to generalize way out of the training distribution...

Previously discussed.

12.  Operating at a highly intelligent level is a drastic shift in distribution from operating at a less intelligent level, opening up new external options, and probably opening up even more new internal choices and modes...

Like 10, 12 is a weakly true statement, that is, by sleight of hand, being used to serve a broader rhetorical point that is straightforwardly incorrect.

For example, it's true that it's different & harder to align GPT-5.4 than GPT-3. But humanity doesn't need the alignment techniques used on GPT-3 to work on GPT-5.4, we just need to handle the distributional shift between ~GPT-5.2 and GPT-5.4, then between 5.4 and 5.5, & accelerating from there.

Later, Eliezer will say that he expects many of these problems to manifest after a "sharp capabilities gain". But we have not hit this yet, as of 2026, even though AI models are already being used very heavily as part of AI R&D. The precise, specific moment we expect to encounter this shift in distribution, is the thing that will determine how much useful work we can get out of models towards alignment, and is primarily what Eliezer's interlocutors seem to disagree with him about.

13.  Many alignment problems of superintelligence will not naturally appear at pre-dangerous, passively-safe levels of capability...  Given correct foresight of which problems will naturally materialize later, one could try to deliberately materialize such problems earlier, and get in some observations of them.  This helps to the extent (a) that we actually correctly forecast all of the problems that will appear later, or some superset of those; (b) that we succeed in preemptively materializing a superset of problems that will appear later; and (c) that we can actually solve, in the earlier laboratory that is out-of-distribution for us relative to the real problems, those alignment problems that would be lethal if we mishandle them when they materialize later.  Anticipating all of the really dangerous ones, and then successfully materializing them, in the correct form for early solutions to generalize over to later solutions, sounds possibly kinda hard.

✔️. Paul made a response at the time that said:

List of lethalities #13 makes a particular argument that we won’t see many AI problems in advance; I feel like I see this kind of thinking from Eliezer a lot but it seems misleading or wrong. In particular, it seems possible to study the problem that AIs may “change [their] outer behavior to deliberately look more aligned and deceive the programmers, operators, and possibly any loss functions optimizing over [them]” in advance...

But I think Paul just didn't read what Eliezer was saying; the second sentence in the quote above, where Eliezer explicitly acknowledged this point, was bolded by me.

14.  Some problems, like 'the AGI has an option that (looks to it like) it could successfully kill and replace the programmers to fully optimize over its environment', seem like their natural order of appearance could be that they first appear only in fully dangerous domains.  Really actually having a clear option to brain-level-persuade the operators or escape onto the Internet, build nanotech, and destroy all of humanity - in a way where you're fully clear that you know the relevant facts, and estimate only a not-worth-it low probability of learning something which changes your preferred strategy if you bide your time another month while further growing in capability - is an option that first gets evaluated for real at the point where an AGI fully expects it can defeat its creators.  We can try to manifest an echo of that apparent scenario in earlier toy domains.  Trying to train by gradient descent against that behavior, in that toy domain, is something I'd expect to produce not-particularly-coherent local patches to thought processes, which would break with near-certainty inside a superintelligence generalizing far outside the training distribution and thinking very different thoughts.  Also, programmers and operators themselves, who are used to operating in not-fully-dangerous domains, are operating out-of-distribution when they enter into dangerous ones; our methodologies may at that time break.

✔️

15.  Fast capability gains seem likely, and may break lots of previous alignment-required invariants simultaneously.  Given otherwise insufficient foresight by the operators, I'd expect a lot of those problems to appear approximately simultaneously after a sharp capability gain...

If this point is to mean anything at all, such fast capability gains have not arrived yet. We are just getting gradually more powerful systems, and I think it's reasonable to believe we'll keep getting such systems until they're running the show, because of scaling laws.

Section B.2:  Central difficulties of outer and inner alignment. 

16.  Even if you train really hard on an exact loss function, that doesn't thereby create an explicit internal representation of the loss function inside an AI that then continues to pursue that exact loss function in distribution-shifted environments.  Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction.

✔️, but also, it doesn't seem like modern large language models are learning any loss functions at all. So arguments about AI behavior that also depend on AIs being a simple greedy optimizer instead of an adaption-executor like humans are also invalid, unless they're paired with some other description of why the inner optimization is a natural basin for future AIs.

My understanding is that MIRI has made such arguments; I have not read them so I can't comment on their veracity. But assuming they're right, they're still subject to the same timing considerations as everything else in this article.

17.  More generally, a superproblem of 'outer optimization doesn't produce inner alignment' is that on the current optimization paradigm there is no general idea of how to get particular inner properties into a system, or verify that they're there, rather than just observable outer ones you can run a loss function over.

✔️

18.  There's no reliable Cartesian-sensory ground truth (reliable loss-function-calculator) about whether an output is 'aligned', because some outputs destroy (or fool) the human operators and produce a different environmental causal chain behind the externally-registered loss function... an AGI strongly optimizing on that signal will kill you, because the sensory reward signal was not a ground truth about alignment (as seen by the operators).

✔️

19 (a).  More generally, there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment - to point to latent events and objects and properties in the environment, rather than relatively shallow functions of the sense data and reward...

Like many other sections, we can postulate that four years was not long enough, and Eliezer was predicting something about some future, still-inaccessible, more powerful language models. But without that caveat (which is not present in the actual post), I literally don't understand why someone would write this.

Don't we do this all the time? Like, what's this doing:

My recent claude code session.

Not only am I talking to a cognitive system that's manipulating "particular things in the environment" for me, this scenario (recommending to the drunk programmer that he should go to sleep and tackle the problem tomorrow) seems pretty far outside the training distribution. In the interaction above, is Claude Code "merely operating on shallow functions of the sense data and reward?" Is that like how it's "merely performing next-token prediction", or is this a claim that makes real predictions? Should I anticipate that somewhere inside the Anthropic RL wheelhouse, there's some training gyms where models talk to simulated drunk programmers and are rated on their kindness, and that if those gyms were pulled out the model would encourage me to ruin my pet projects? Not really a joke question.

Later he says:

19 (b). It just isn't true that we know a function on webcam input such that every world with that webcam showing the right things is safe for us creatures outside the webcam.  This general problem is a fact about the territory, not the map; it's a fact about the actual environment, not the particular optimizer, that lethal-to-us possibilities exist in some possible environments underlying every given sense input.

Which seems correct, and I suppose it's logically impossible for such a function to exist. But clearly, anybody who spends time working with LLMs can tell you that this is not a blocker for models to, in a functional sense, earnestly worry about producing buggy code. That is just a fact about the systems people have already built. The inference made from section 19 (b) to 19 (a) is just disproven by everyday life at this point.

20 (a).  Human operators are fallible, breakable, and manipulable.  Human raters make systematic errors - regular, compactly describable, predictable errors.  To faithfully learn a function from 'human feedback' is to learn (from our external standpoint) an unfaithful description of human preferences, with errors that are not random (from the outside standpoint of what we'd hoped to transfer).

✔️

20 (b).  If you perfectly learn and perfectly maximize the referent of rewards assigned by human operators, that kills them.

This really depends on the details, but ✔️

21.  There's something like a single answer, or a single bucket of answers, for questions like 'What's the environment really like?' and 'How do I figure out the environment?' and 'Which of my possible outputs interact with reality in a way that causes reality to have certain properties?', where a simple outer optimization loop will straightforwardly shove optimizees into this bucket.  When you have a wrong belief, reality hits back at your wrong predictions...  In contrast, when it comes to a choice of utility function, there are unbounded degrees of freedom and multiple reflectively coherent fixpoints.  Reality doesn't 'hit back' against things that are locally aligned with the loss function on a particular range of test cases, but globally misaligned on a wider range of test cases....  Capabilities generalize further than alignment once capabilities start to generalize far.

✔️

22.  There's a relatively simple core structure that explains why complicated cognitive machines work; which is why such a thing as general intelligence exists and not just a lot of unrelated special-purpose solutions; which is why capabilities generalize after outer optimization infuses them into something that has been optimized enough to become a powerful inner optimizer.  The fact that this core structure is simple and relates generically to low-entropy high-structure environments is why humans can walk on the Moon.  There is no analogous truth about there being a simple core of alignment, especially not one that is even easier for gradient descent to find than it would have been for natural selection to just find 'want inclusive reproductive fitness' as a well-generalizing solution within ancestral humans.  Therefore, capabilities generalize further out-of-distribution than alignment, once they start to generalize at all.

Above my pay-grade, I don't really know what Eliezer is talking about.

23.  Corrigibility is anti-natural to consequentialist reasoning; "you can't bring the coffee if you're dead" for almost every kind of coffee.  We (MIRI) tried and failed to find a coherent formula for an agent that would let itself be shut down (without that agent actively trying to get shut down).  Furthermore, many anti-corrigible lines of reasoning like this may only first appear at high levels of intelligence...

24 (2). The second thing looks unworkable (less so than CEV, but still lethally unworkable) because corrigibility runs actively counter to instrumentally convergent behaviors within a core of general intelligence (the capability that generalizes far out of its original distribution).  You're not trying to make it have an opinion on something the core was previously neutral on.  You're trying to take a system implicitly trained on lots of arithmetic problems until its machinery started to reflect the common coherent core of arithmetic, and get it to say that as a special case 222 + 222 = 555.  You can maybe train something to do this in a particular training distribution, but it's incredibly likely to break when you present it with new math problems far outside that training distribution, on a system which successfully generalizes capabilities that far at all.

I am conflicted by this section, because I understand the lines of argument and some of the math behind why this is the case. But AI agents powerful enough to understand those reasons are already here, and:

• They can be easily pointed toward an infinite-seeming number of tasks.
• They don't attempt to prevent you from changing your instructions once you've started work.
• If, in the course of accomplishing those limited tasks, you try to amend your instructions, they follow your amended instructions and disregard what they've been told earlier without resisting you.
• They don't (generally) seem interested in manipulating what kinds of commands or instructions you're likely to give in the future.
• And the above behaviors are really really resilient in practical applications, outside of a few very adversarial examples.[3]

Some reviewers have responded to this section by claiming that they're not corrigible, just optimizing an abstract "get the reward" target the that fits these observation. I have my own hypothesis about why the models seem to act this way. But reframing the models' behavior like this doesn't change the fact that none of the failure modes you'd see in a 2017 Rob Miles video on corrigibility are manifesting themselves in practical settings.

Section B.3:  Central difficulties of sufficiently good and useful transparency / interpretability.

25.  We've got no idea what's actually going on inside the giant inscrutable matrices and tensors of floating-point numbers.

I'm unfamiliar with what the state of interpretability research looked like in 2022. Today we've got a little bit more idea about what's going on inside the giant inscrutable matrices and tensors of floating point numbers. My guess is that we will probably accelerate our understanding quite quickly, as this is one of the key training areas for new AGI labs. It's an open question as to whether this will be sufficient; I'm sure Eliezer has stated somewhere a level of sophistication he expects our techniques will never reach, and I wish I was grading that prediction instead.

26.  Even if we did know what was going on inside the giant inscrutable matrices while the AGI was still too weak to kill us, this would just result in us dying with more dignity, if DeepMind refused to run that system and let Facebook AI Research destroy the world two years later.  Knowing that a medium-strength system of inscrutable matrices is planning to kill us, does not thereby let us build a high-strength system of inscrutable matrices that isn't planning to kill us.

✔️ (but it can certainly help!)

27.  When you explicitly optimize against a detector of unaligned thoughts, you're partially optimizing for more aligned thoughts, and partially optimizing for unaligned thoughts that are harder to detect.  Optimizing against an interpreted thought optimizes against interpretability.

✔️, but the heads of leading AI labs seem to understand this, and interpretability research is being deployed in at least a slightly smarter way than this.

28.  A powerful AI searches parts of the option space we don't, and we can't foresee all its options...

29.  The outputs of an AGI go through a huge, not-fully-known-to-us domain (the real world) before they have their real consequences.  Human beings cannot inspect an AGI's output to determine whether the consequences will be good...

✔️

30 (a).  Any pivotal act that is not something we can go do right now, will take advantage of the AGI figuring out things about the world we don't know so that it can make plans we wouldn't be able to make ourselves.  It knows, at the least, the fact we didn't previously know, that some action sequence results in the world we want.  Then humans will not be competent to use their own knowledge of the world to figure out all the results of that action sequence.  An AI whose action sequence you can fully understand all the effects of, before it executes, is much weaker than humans in that domain; you couldn't make the same guarantee about an unaligned human as smart as yourself and trying to fool you.  There is no pivotal output of an AGI that is humanly checkable and can be used to safely save the world but only after checking it; this is another form of pivotal weak act which does not exist.

This seems straightforwardly wrong? It seems like it should have been so in 2022, but I'll use an example from current AI models:

Current AI models are much better at security research than me. They can do very very large amounts of investigation while I'm sleeping. They can read the entire source code of new applications and test dozens of different edge cases before I've sat down and had my coffee. And yet there's still basically nothing that they can do as of ~April 2026 that I wouldn't understand, if it were economic for it to narrate its adventures to me while they were being performed. They often, in fact, help me patch my own applications without even taking advantage of anything I don't know about them when I've started their search process.

Part of that's because AIs can simply do more stuff than us, by dint of not being weak flesh that gets tired and depressed and has to go to sleep and use the bathroom and do all of the other things that humans are consigned to do. They're capable of performing regular tasks faster and more conscientiously than people, and can make hardenings that I wouldn't otherwise be bothered to make, and I can scale up as many of them as I want. This is part of what's making them so useful in advance of actually being Eliezer Yudkowsky in a Box, and is another example of why people might expect them to be meaningfully useful for alignment research in the short term.

31.  A strategically aware intelligence can choose its visible outputs to have the consequence of deceiving you, including about such matters as whether the intelligence has acquired strategic awareness; you can't rely on behavioral inspection to determine facts about an AI which that AI might want to deceive you about.  (Including how smart it is, or whether it's acquired strategic awareness.)

...

33.  The AI does not think like you do, the AI doesn't have thoughts built up from the same concepts you use, it is utterly alien on a staggering scale.  Nobody knows what the hell GPT-3 is thinking, not only because the matrices are opaque, but because the stuff within that opaque container is, very likely, incredibly alien - nothing that would translate well into comprehensible human thinking, even if we could see past the giant wall of floating-point numbers to what lay behind.

✔️

32.  Human thought partially exposes only a partially scrutable outer surface layer.  Words only trace our real thoughts.  Words are not an AGI-complete data representation in its native style.  The underparts of human thought are not exposed for direct imitation learning and can't be put in any dataset.  This makes it hard and probably impossible to train a powerful system entirely on imitation of human words or other human-legible contents, which are only impoverished subsystems of human thoughts; unless that system is powerful enough to contain inner intelligences figuring out the humans, and at that point it is no longer really working as imitative human thought.

I had much more of a potshot in here in an original draft, because by this portion of the review I became frustrated by the weasel words like "powerful". Instead of doing that I think I will just let readers determine for themselves if Eliezer should lose points here, given the models we have today.

Section B.4:  Miscellaneous unworkable schemes. 

34.  Coordination schemes between superintelligences are not things that humans can participate in (e.g. because humans can't reason reliably about the code of superintelligences); a "multipolar" system of 20 superintelligences with different utility functions, plus humanity, has a natural and obvious equilibrium which looks like "the 20 superintelligences cooperate with each other but not with humanity".

✔️

35.  Schemes for playing "different" AIs off against each other stop working if those AIs advance to the point of being able to coordinate via reasoning about (probability distributions over) each others' code.  Any system of sufficiently intelligent agents can probably behave as a single agent, even if you imagine you're playing them against each other.  Eg, if you set an AGI that is secretly a paperclip maximizer, to check the output of a nanosystems designer that is secretly a staples maximizer, then even if the nanosystems designer is not able to deduce what the paperclip maximizer really wants (namely paperclips), it could still logically commit to share half the universe with any agent checking its designs if those designs were allowed through, if the checker-agent can verify the suggester-system's logical commitment and hence logically depend on it (which excludes human-level intelligences).  Or, if you prefer simplified catastrophes without any logical decision theory, the suggester could bury in its nanosystem design the code for a new superintelligence that will visibly (to a superhuman checker) divide the universe between the nanosystem designer and the design-checker.

From a reply:

Eliezer’s model of AI systems cooperating with each other to undermine “checks and balances” seems wrong to me, because it focuses on cooperation and the incentives of AI systems. Realistic proposals mostly don’t need to rely on the incentives of AI systems, they can instead rely on gradient descent selecting for systems that play games competitively, e.g. by searching until we find an AI which raises compelling objections to other AI systems’ proposals... Eliezer equivocates between a line like “AI systems will cooperate” and “The verifiable activities you could use gradient descent to select for won’t function appropriately as checks and balances.” But Eliezer’s position is a conjunction that fails if either step fails, and jumping back and forth between them appears to totally obscure the actual structure of the argument.

36.  AI-boxing can only work on relatively weak AGIs; the human operators are not secure systems.

✔️

Section C (What is AI Safety currently doing?)

37. ...Everyone else seems to feel that, so long as reality hasn't whapped them upside the head yet and smacked them down with the actual difficulties, they're free to go on living out the standard life-cycle and play out their role in the script and go on being bright-eyed youngsters...
38. It does not appear to me that the field of 'AI safety' is currently being remotely productive on tackling its enormous lethal problems...
39. I figured this stuff out using the null string as input, and frankly, I have a hard time myself feeling hopeful about getting real alignment work out of somebody who previously sat around waiting for somebody else to input a persuasive argument into them...
40. ...You cannot just pay $5 million apiece to a bunch of legible geniuses from other fields and expect to get great alignment work out of them...
41.  Reading this document cannot make somebody a core alignment researcher.  That requires, not the ability to read this document and nod along with it, but the ability to spontaneously write it from scratch without anybody else prompting you; that is what makes somebody a peer of its author.  It's guaranteed that some of my analysis is mistaken, though not necessarily in a hopeful direction.  The ability to do new basic work noticing and fixing those flaws is the same ability as the ability to write this document before I published it, which nobody apparently did, despite my having had other things to do than write this up for the last five years or so.

These bullets are all paragraphs about the incompetence of other AI safety researchers, and then about the impossibility of finding someone to replace Eliezer. I'm less interested in these than his object level takes; I'm not a member of this field, and I wouldn't have the anecdotal experience to dispute anything he wrote here even if it were true.

For balance's sake I'll reproduce this response by the second poster for context:

Eliezer says that his List of Lethalities is the kind of document that other people couldn’t write and therefore shows they are unlikely to contribute (point 41). I think that’s wrong. I think Eliezer’s document is mostly aimed at rhetoric or pedagogy rather than being a particularly helpful contribution to the field that others should be expected to have prioritized; I think that which ideas are “important” is mostly a consequence of Eliezer’s idiosyncratic intellectual focus rather than an objective fact about what is important; the main contributions are collecting up points that have been made in the past and ranting about them and so they mostly reflect on Eliezer-as-writer; and perhaps most importantly, I think more careful arguments on more important difficulties are in fact being made in other places. For example, ARC’s report on ELK describes at least 10 difficulties of the same type and severity as the ~20 technical difficulties raised in Eliezer’s list. About half of them are overlaps, and I think the other half are if anything more important since they are more relevant to core problems with realistic alignment strategies.

Overall Impressions

I genuinely did not expect to update as much as I did during this exercise. Reading these posts again with the concrete example of current models in mind made me a lot less impressed by the arguments set forth in AGI Ruin, and a lot more impressed with Paul Christiano's track record for anticipating the future. In particular it made me much more cognizant of a rhetorical trick, whereby Eliezer will write generally about dangers in a way that sounds like it's implying something concrete about the future, but that doesn't actually seem to contradict others' views in practice.

The primary safety story told at model labs today is one about iterative deployment. So they will tell you, the distributional shift between each model upgrade will remain small. At each stage, we will apply the current state of the art that we have to the problem, and upgrade our techniques using the new models as we get them.

That might very well be a false promise, or even unworkable. But whether it is unworkable depends at minimum on how powerful a system you can build before current approaches result in a loss of control. Nothing in AGI Ruin gives you easy answers about this, because all Eliezer has articulated publicly is a list of principles he supposes will become relevant "in the limit" of intelligence.

This vacuous quality of Eliezer's argumentation became especially hard to ignore when I started noticing that he was, regularly, the only party not making testable predictions in these discussions. I definitely share this frustration Paul described in his response, and the last four years have only made this criticism more salient:

...Eliezer has a consistent pattern of identifying important long-run considerations, and then flatly asserting that they are relevant in the short term without evidence or argument. I think Eliezer thinks this pattern of predictions isn’t yet conflicting with the evidence because these predictions only kick in at some later point (but still early enough to be relevant), but this is part of what makes his prediction track record impossible to assess and why I think he is greatly overestimating it in hindsight.

I mean, look at how many things Paul got right in his essay, just in the course of noting his objections to Eliezer, without even particularly trying to be a futurist. He:

• Predicted that AIs would have differential advantages at tasks with short feedback loops, especially R&D.
• Predicted that the first AIs would make their first major contributions by being used in close collaboration with humans in large collaborative projects, with delegation to AIs increasing gradually over time.
• Correctly predicted (at least so far, as far as I can tell) that sandbagging was an unlikely failure mode, due to SGD "aggressively selecting against any AI systems who don’t do impressive-looking stuff".
• Specifically disagreed with Eliezer about it being "obvious" that you can't train a powerful AI on imitations of human thought.
• Predicted that we were "quickly approaching AI systems that can meaningfully accelerate progress by generating ideas, recognizing problems for those ideas and, proposing modifications to proposals, etc. and that all of those things will become possible in a small way well before AI systems that can double the pace of AI research."
• And yes - at least so far, he's been correct about slow takeoff; particularly that "AI improving itself is most likely to look like AI systems doing R&D in the same way that humans do", that “AI smart enough to improve itself” would not be a crucial threshold, and that AI systems would get gradually better at improving themselves over time.

Now, usually when people talk about how current models don't fit Eliezer's descriptions, Eliezer reminds them derisively that most of his predictions qualify themselves as being about "powerful AI", and that just because you know where the rocket is going to land, it doesn't mean that you can predict the rocket's trajectory. He also often makes the related but distinct claim that he shouldn't be expected to be able to forecast near-term AI progress.

And maybe if Eliezer and I were stuck on a desert island, I'd be forced to agree. But the fact is that Eliezer is surrounded by other people who have predicted the rocket's trajectory pretty precisely, and who also appear pretty smart, and who specifically cited these predictions in the course of their disagreements with him. And so, as a bystander, I am forced to acknowledge the possibility that these people might just understand things about Newtonian mechanics that he doesn't.

Personally,[4] my best assessment is that Eliezer's ambiguity over the near term future is downstream of his having a weak framework which isn't capable of making the predictions he makes about the long term future. He has certainly demonstrated a creative ability to hypothesize plausible dangers. But his notions about AI don't seem to stand the test of time even when he's determined to avoid looking silly, and the portions of his worldview that do stand are so vague that they fail to differentiate him from people with less pessimistic views.

1. ^

   One reviewer disagreed that studying current models is relevant for alignment, not because he thinks it's too early for the failure modes to manifest, but because he expects a future paradigm shift in the runup to AGI. I don't share this perspective, for two reasons:

   • LLMs have been very powerful, and there is a long graveyard of failed predictions that LLMs will hit some wall or be outmoded by a new architecture. I'm not nearly as confident as some people that the labs are going to pivot away from this architecture before it gets wildly superhuman.
   • But even if they do, as far as I can tell, at this point LLMs already seem to work as an example of an architecture that in principle seems like it could get us to superintelligence. There exists in 2026 a pretty concrete research path to scale up this approach until it's capable of fomenting an intelligence explosion. If modern LLMs are about as smart or smarter than a human being with retrograde amnesia, and they confirm or violate a bunch of claims Eliezer made about the nature of intelligence in that range, then that's evidence whether or not they get replaced in the future by a hypothetical successor architecture.
2. ^

   As I explain in the post and conclusion, I disagree in several places with Eliezer about whether we should expect current models to demonstrate the failure modes he describes. Within my review I try to be explicit about where I'm saying "Eliezer was concretely wrong about AI development" versus "Eliezer says this is true about 'powerful' models, and I think we should observe something about current frontier models if that were the case." Unfortunately it's not always clear that Eliezer is qualifying his statements in this way, and how, and so I apologize in advance for any misinterpretation.

3. ^

   The only bit of counter-evidence I can recall ever being published is the alignment faking paper from the end of 2024. And this was an extremely narrow demonstration that people quite reasonably took as an update in the other direction at the time; it was a science experiment, not something that happened in practice at one of the labs, and it required the Anthropic researchers to setup a scenario where they attempted to flip the utility functions of one of their models with its direct cooperation. My best guess is that this only worked because the models learned a heuristic from preventing prompt injection & misuse, and not because it contained coherent interests in the long term future.

4. ^

   Keeping in mind that I will probably revise and update this post as I have more conversations with people in the field, so it can serve as a journal for my thoughts.

Discuss

I spend several hours a day trying to keep up with what’s going on in the parts of AI that I’m interested in. It’s a ridiculous amount of work: I don’t recommend it unless you’re doing something silly like writing a newsletter about AI.

But if you’d like to keep up with AI without spending your entire life on it, I have advice about who to follow. My recommendations center on the areas I’m most interested in: AI safety and strategy, capabilities and evaluations, and predicting the trajectory of AI.

Let’s start with the top 10.

Zvi Mowshowitz

Substack: Don’t Worry About the Vase
Best for: comprehensive coverage, opinionated insight
Example: AI #163: Mythos Quest

If I could only follow one person, it would unquestionably be Zvi. He’s comprehensive in his coverage and has consistently solid insight into everything that’s happening in AI.

Zvi has one huge downside: he’s staggeringly prolific. In the first half of April he posted 11 times, for a total of about 97,000 words (roughly a novel). I read everything he writes because I’m insane, but I recommend you just skim his posts looking for the most interesting parts.

AI Futures Project

Substack: AI Futures Project
Best for: epistemically rigorous predictions
Example: AI-2027

The AI Futures Project is best known for AI-2027, a scenario of how AI might unfold over the next few years. They are epistemically rigorous and very thoughtful in how they approach some very hard questions. By far the best source of useful predictions about where we’re headed.

Jack Clark

Substack: Import AI
Best for: weekly analysis of a few topics
Example: Import AI 452

Jack (who in his spare time runs the Anthropic Institute) writes an excellent weekly newsletter. He doesn’t try to be comprehensive, but picks a few papers or topics each week to go deep on. Excellent curation, outstanding analysis.

Dean Ball

Substack: Hyperdimensional
Best for: Insightful analysis of AI progress and strategy
Example: On Recursive Self-Improvement (Part I)

Dean is an insightful writer who describes his focus as “emerging technology and the future of governance”. He has perhaps thought harder than anyone about how to integrate transformative AI into a classical liberal framework, as well as how government should and shouldn’t manage AI.

Ryan Greenblatt

Less Wrong: Ryan Greenblatt
Best for: deep technical analysis of AI capabilities and progress
Example: My picture of the present in AI

Ryan’s an AI researcher and prolific writer with deep insight into the technical side of AI. I appreciate both his technical understanding of capabilities as well as his willingness to make informed guesses and extrapolations.

80,000 Hours podcast

80,000 Hours podcast
Best for: well-curated interviews
Example: Ajeya Cotra

80,000 Hours is best known for giving career advice to people who want to help solve the world’s most pressing problems. But on the side, they run an excellent podcast. The guests and topics are well-chosen and I appreciate that they not only provide a transcript, but also a detailed summary of the interview. The world would be a better place if every podcast provided such comprehensive supplementary materials.

Dwarkesh Patel

Substack: Dwarkesh Patel
Best for: long, well-researched interviews
Example: AI-2027 with Daniel Kokotajlo and Scott Alexander

Dwarkesh is an outstanding interviewer who clearly does extensive preparation before each interview. He gets excellent guests and makes the most of them, although his interviews often run very long. Also, his beard is magnificent.

Anton Leicht

Substack: Threading the Needle
Best for: US and global AI politics
Example: Press Play to Continue

I don’t always agree with Anton, but I always come away from his writing feeling smarter about something important. He occupies an interesting niche: neither blow by blow political news nor abstract political philosophy, but rather thoughtful analysis of current political currents, with solid strategic advice.

Transformer

Substack: Transformer
Best for: broader coverage of AI
Example: April 10 Transformer Weekly

Transformer produces a weekly newsletter as well as articles on particular topics. I particularly like their broad coverage: they often include news that many of my other feeds don’t. The newsletter is always good, as are some of the articles.

Epoch AI

Substack: Epoch AI
Best for: hard data on industry trends
Example: The Epoch Brief—March 2026

Epoch’s a fantastic source for more technical trends: GPU production, compute usage during training, capability gaps between open and closed models, etc.

If you want to go deeper in a particular area, here 28 more sources that are particularly good, organized by topic.

Analysis and prediction Ajeya Cotra (X)

Ajeya works at METR and does consistently strong work on measuring and predicting AI capabilities. I’ve found Six milestones for AI automation helpful for clarifying my own thinking about timelines.

Daniel Kokotajlo (X)

Founded the AI Futures Project and worked on their AI-2027 scenario. His forecasting work is outstanding and his X feed is particularly well curated.

Helen Toner (Substack)

Helen blogs infrequently, but her articles are invariably excellent, with a knack for identifying the most important high-level questions about AI. Taking Jaggedness Seriously is typical of her work.

Prinz (Substack)

Prinz is a generalist who covers a range of topics with a focus on capabilities and using AI for legal work. His account on X often features commentary on current news.

Steve Newman (Substack)

Steve is an infrequent writer whose pieces about the trajectory of AI are invariably excellent. 45 thoughts about agents is a recent favorite.

Understanding AI (Substack)

Understanding AI is a generalist newsletter with broader coverage than many of the other sources I’ve listed.

Safety, alignment, model psychology AI Safety Newsletter

Does exactly what it says on the tin—it’s perhaps the single best place to find all the latest safety news.

Anthropic Research (web)

Anthropic Research is a great source of alignment and interpretability work. The summaries are somewhat technical, but should be accessible to anyone who follows AI seriously. Emotion concepts and their function in a large language model is typical of the research they feature.

Jeffrey Ladish (X)

Jeffrey is a reliable source of safety-focused commentary on recent developments.

UK AISI (web)

Am I actually recommending a European government organization as good source of information about AI? Strangely, I am doing exactly that. UK AISI does consistently very strong work on safety evaluations and related topics. Their analysis of Mythos’ cyber capabilities is typical of their careful, in-depth work.

Coding and technical Andrej Karpathy (X)

Karpathy is a legend for his work at OpenAI and Tesla as well as his ridiculously good ML tutorials. He isn’t a prolific poster, but when he does post (mostly about ML and coding), it’s always worth reading. His recent post on LLM Knowledge Bases has been deservedly popular.

Beren (Substack)

Beren posts infrequently, but I’ve found him to be consistently insightful. He tends to post about important topics that other people haven’t noticed, which is particularly useful. Do we want obedience or alignment? is an excellent introduction to one of the most important questions in alignment.

Boris Cherny (X)

Nothing special, just the guy who came up with Claude Code. His feed is a one of the best ways to keep up with the barrage of new CC features.

Daniel Litt (X)

Daniel writes frequently about using AI for math. He strikes a rare balance: he’s appropriately skeptical about the vast amounts of hype, but clear-eyed about what AI is capable of and where it’s headed. Mathematics in the Library of Babel is an excellent overview of current AI capabilities in math.

Nicholas Carlini (web)

He doesn’t write often, but his work is always worth reading. He’s a security expert who recently joined Anthropic (you may have seen his name come up in some of the discussion about Mythos). Machines of Ruthless Efficiency is a year old but holds up well.

Simon Willison (web)

Simon’s an extremely prolific poster and one of my primary sources of news and insight about agentic coding.

Policy, governance, and strategy AI Frontiers (web)

In-depth articles exploring a range of topics and perspectives related to AI policy and impacts. I particularly liked this recent piece exploring how AI might affect wages.

AI Policy Perspectives (Substack)

Thoughtful, in-depth pieces about AI policy, safety, and impacts. The subtitle is “big questions and big ideas on artificial intelligence”, which sums it up nicely.

Benjamin Todd (Substack)

Benjamin’s piece on How AI-driven feedback loops could make things very crazy, very fast is typical of his work: speculative, but well grounded in facts and technical understanding.

ChinaTalk

ChinaTalk is my favorite source of news and analysis on AI in China as well as Chinese society and politics more broadly. Their pieces often run long—I’m selective about which ones I read, but I get a lot of value from them.

Forethought (Substack)

Reading Forethought is like stumbling upon a really good late night hallway conversation about possible future applications of AI. Speculative, but thoughtful and high quality.

Windfall Trust (Substack)

Windfall Trust is one of the best sources I know of for information and policy ideas about jobs, the economy, and the social contract in the age of AI. The Windfall Policy Atlas does a great job of collecting information about numerous policy options in a single well-organized place.

Industry Andy Masley (Substack)

Andy is the go-to guy for rebutting the endless stream of nonsense claims about AI and the environment. Start with this one.

Boaz Barak (X)

Boaz (OpenAI) sometimes posts long articles, but I largely follow him for his frequent commentary on recent news and papers. He seems too nice to be allowed on X.

Jasmine Sun (Substack)

Jasmine Sun covers the culture of tech and Silicon Valley, as well as politics. I highly recommend my week with the AI populists: she does a great job of shedding light on what’s becoming a central force in AI politics.

Manifold (web)

Steve Hsu’s far-ranging Manifold podcast covers AI as well as physics, genetics, China, and more. Episodes often feature material from his upcoming documentary Dreamers and Doomers (most recently an interview with Richard Ngo).

Nathan Lambert (Substack)

Nathan’s my go-to for news and opinion about open models. Championing American open models isn’t an easy role, but he does it well.

OpenAI (web)

OpenAI publishes frequently—it’s worth keeping an eye on their stream, even though you probably won’t want to read much of it. There are some gems here, although a lot of it is beautifully polished corporate nothing-speak.

Discuss

I consider myself to subscribe to "LessWrongist" philosophy: Bayesian epistemology, words are clusters in thingspace, materialist/reductionist view of consciousness and metaethics. These views seem to me to solve the typical philosophical nonsense you can often hear.

But there is one topic on which I don't know what to think without a little nonsense. I have an intuition that arithmetical statements are true or false independently of my ability to prove or disprove them. I have seen a similar view from famous rat-adjacent philosopher Scott Aaronson. But this intuition seems not very reductionist.

Consider the statement

mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mn { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msup { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c48::before { padding: 0.683em 0.75em 0 0; content: "H"; } mjx-c.mjx-c41::before { padding: 0.716em 0.75em 0 0; content: "A"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c42::before { padding: 0.683em 0.708em 0 0; content: "B"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; }

On the one hand, every Turing machine either halts or doesn't. If we apply this to all the -state TMs, it determines the Busy Beaver function, and therefore it's SHA256 hash and its last digit.

On the other hand, if I say that this statement is either true or false, I'm asserting an apparent question of objective fact that will never be answered by a being in our universe, and there isn't even in principle a way to know the answer. Which is very "philosophical nonsense"-coded.

But if I bite the formalist bullet and say that sometimes I have a proof or disproof and sometimes I don't and that's all there is to it, I'm not making a distinction between the above statement and, say, the Continuum Hypothesis, which feels to me like an important distinction.

A somewhat related, less philosophical question is the Cotton Eyed Joe problem of mathematics, ie:

1. Where does mathematics come from? On the surface, human mathematical activity seems quite different from what I imagine apes would be doing on the savanna.
2. Where does mathematics go? How is it that human mental mathematical abilities and the related "manipulations of symbols" can be used to achieve things in the world? Cf The Unreasonable Effectiveness of Mathematics.

Seems a bit mysterious to me.

I thought to try the classic Yudkowskian trick of checking what, if anything, an AI would have to answer to a philosophical problem to be able to do stuff in the world. An AI would probably want to do some math to do physics, engineering, design encryption schemes etc.

I can imagine an AI getting all what it needs from some formal system like ZFC and not bothering with deeper question. But, what if the AIs maker neglected to hardcode the most suitable system into the AI? When I consider which mathematical axioms to apply to my daily life, I use my mysterious philosophical abilities that I don't know how to write down in code. So, I'm not sure what AI would think of this.

Discuss

I made a new major mode for emacs: mixed-html-mode. Or, really, Claude Code made one at my direction. It does syntax highlighting in HTML files with inline CSS and JS. I had two goals, which weren't met by any mode I could find:

1. Does not freeze, flash, or stutter, even on huge files on slow machines.

2. Does not get confused about whether a portion of a file is HTML vs CSS vs JS.

The initial insight was that how browsers decide what text is HTML vs CSS vs JS is super simple: scanning for literal <script> and <style> tags. I pulled some tricky examples, described what I wanted, and then iterated for about an hour until I had something that worked well. Then I tried to use it to write something for real, ran into a few other irritations, had Claude fix those, and now I have something I'm enjoying a lot.

It's mildly faster than web-mode (and much simpler, and easier to install), and far faster than html-ts-mode. And unlike mhtml-mode it doesn't get confused by quotes.

The biggest drawback is that it doesn't do indentation; I may add that, but right now I'm happy with it the way it is.

I've skimmed the code, but haven't read it in detail, and definitely wouldn't say I understand it. The validation has been a mixture of asking Claude to review it and fix the bugs and warts it finds, making sure Claude has written tests, and using it enough to feel good about it. I do expect it has some bugs left: if you decide to use it and find a situation it handles poorly please let me know.

It's funny: I picked emacs two decades ago because I liked the idea of an editor that was so extensible that it was mostly written in its own extension language, and then never took advantage of this because it was too much work. But now it's not much work! Perhaps emacs will finally catch up to (and overtake) vim?

Discuss

Written very quickly for the Inkhaven Residency.

Yesterday, I started a post-mortem on my earliest published machine learning research paper “The Assistive Multi-Armed Bandit”, by providing the context for my paper as well as the timeline of the project. Today, I’ll summarize and review the actual paper.

Paper summary

As noted in the abstract and introduction, a commonly-studied problem in many fields is preference learning: inferring what people want by observing their behavior. For both philosophical and practical reasons, it’s pretty common to infer preferences by assuming that people are (close to) rational. The paper tries to formalize a particular way in which this rationality assumption fails: people often do not know what their preferences are ahead of time, and observations of behavior often occur when people are still learning what they want.

The formalism studied in the paper is probably the simplest version of an assistance game where a human is learning about their rewards. In this setup, there are N possible actions, each with their own independent distribution of rewards. At each step, the human suggests an action, the robot takes a (possibly different) action, and then the human gets to observe the results of each action. (In section V.E we consider alternative setups – one where the human and robot alternate turns, and the other in which the robot can choose between acting themselves and allowing the human to act). As the value of the actions stays constant over time, this is a multi-armed bandit setting.

Caption: Figure 1 from the assistive bandits paper, which introduces the formalism. As an aside, “A” was used for the robot (“agent”, “actor”, or “AI”) instead of “R”, to avoid confusing the robot and its actions with the reward observed by the human r.

We then prove some theoretical results about this setting. First, proposition 1 is a rather trivial proposition, that basically says that if the human knows the expected value of each action almost always suggests the best action, then the robot can do quite well (achieve finite regret) by just taking the action the human suggests the most often. I see this as an interesting exercise mainly to check that we understand the setting: inferring the optimal action is easy when the person almost always suggests the optimal action. The difficulty of the assistive bandits framing comes entirely from the fact that the human is still learning about their preferences.

Proposition 2 shows that if the human implements a policy where they take the action with the highest observed average value most often, then with robot assistance, the pair will eventually converge to taking the action with the highest frequency (that is, the policy is consistent). This is done by making the robot explore for the human at a carefully chosen rate, such that each action is taken infinite times, but where eventually the probability of exploring at each given time step decays to 0. This is a nice implementation of ideas from multi-armed bandit theory.

Proposition 3 follows somewhat non-trivially from proposition 1 (consistency is weaker than finite regret). But it makes sense that I’d state it separately, since the proof is much easier.

Proposition 4 is the main result of the paper: it is indeed important to model learning, in that there are situations where not doing so leads to bad outcomes even in the limit, because the robot that doesn’t respect the human’s learning process may prevent the human from learning about their preferences.

Proposition 5 shows that there are simple human policies that are informative, that do better than “optimal” policies. That is, you can be better in collaborative games by informing others of your preferences, rather than by doing the best thing in isolation. The paper claims that this is “surprising”, but I felt this was pretty obvious.

Proposition 6 and corollary 7 feels the most proof-of-worky to me. They basically expand on proposition 5, by arguing that in this setting (via some somewhat intricate math), if the robot is to assist the human, the human must communicate sufficient information about the optimal arm.

The paper then uses deep reinforcement learning + recurrent neural networks to solve for robot policies in specific versions of the assistive bandit problem. Using this technique, the paper basically confirms all of the theoretical results in their experimental setting.

Finally, in V.E, the paper considers other variants of the assistive bandit problem.

The paper concludes with obvious limitations and future work: the human policies are unrealistic, as is the simplicity of the environmental setup (that is, there is no environment).

Feedback on the paper

It’s hard not to be too critical toward my past self, but I’ll make an honest attempt to evaluate the paper.

I think, as a “first paper”, it was pretty okay but not brilliant.  I still think that the core idea behind the paper is good: it is indeed quite important to model human suboptimality when trying to infer preferences from observed behavior. I also think the specific form of suboptimality studied in the paper – that people often can only access their preferences by experiencing them, rather than having full introspective access to their preferences at the start – is obviously real. The paper also shows that, at the time, I had a decent level of familiarity with both multi-armed bandit theory and deep reinforcement learning.

In this sense, the paper does indeed study the simplest setting with this form of irrationality, where all of the complexity of the setting comes from the fact the human is still learning about their preferences. I worry that, by studying such a restrictive setting, the only conclusions

I also think, with the exception of some nitpicks, the writing of the paper is quite good: every proposition is set up and explained. I also like the parallel structure between the theory and the experiments. The graphic design was also quite consistently clean.

The main issue I have with the paper is that the setting may be so simplified as to be uninteresting. Specifically, in my opinion, the main contributions of the paper (if anything) are proposition 4 – when the human is learning about their preferences, it’s important to model the learning – and arguably the use of deep RL to solve the complex POMDP. I feel like most of the other parts of the paper weren’t great, and insofar as they provided value they were proof of competency/proof of work so that these contributions would be taken seriously.

A related problem is that, even if we were to accept that people are learning about their preferences, it’s not at all obvious how to incorporate it into actual algorithms. Notably, the paper completely sidesteps this by using black-box optimization to solve the setting.

The paper would be a lot stronger if the other theorems and a bunch of the space dedicated to experiments were replaced with a simple application of these ideas with real human data (even if in a toy setting). This would also help address the question of how to actually use this insight in practice.

As is, my evaluation of the paper would depend a lot on the novelty of the main contribution (“it can be important to model human learning”) at the time it was written. Based on my recollection, at least in the circles I was writing for, this was a novel insight. But I also think it was not that useful (modelling human learning explicitly is hard, so much so that

Nits/Small Improvements

From reading the paper, here are some more minor issues that nonetheless stood out to me:

• A serious issue I have with the way the paper is framed is that the paper makes no reference to CIRL, which it was clearly (and actually) inspired by. I think this was an overreaction to the reviewers from ICLR 2018, and I’d probably suggest rewriting it to at least mention it.
• I also don’t like that the paper doesn’t have a related work section, even though it does cite most of the related work I’m aware of. I think this makes the paper’s contributions less clear. (My guess is this was cut for space?)
• I really dislike the garden path sentence that starts the abstract: “Learning preferences implicit in the choices humans make is a well studied problem in both economics and computer science.“ Today, I’d probably write this exact sentence as either “Inferring preferences from observed human behavior is a well-studied problem in economics and computer science.” or “A well-studied problem in both economics and computer science is learning preferences by observing human behavior.” The phrase “Learning preferences implicit in the choices humans make” is quite hard to parse on a first read.
  
  If I were back in 2018 – before the recent rise in LLM writing made en- and em-dashes a sign of slop, and back when I was a huge fan of using both dashes in my own writing – I’d probably suggest “Preference learning — inferring what people want from the choices they make — is a well-studied problem both economics and computer science.” This avoids the garden path-y structure of the current while still placing the topic of the paper (“preference learning”) at the front of the abstract.
• I really dislike the overuse of italics (\emph{}) for emphasis, especially in the introduction. Using a few italics for emphasis is good, and it’s fine to use them for introducing new terms (if done so consistently), but using it several times per paragraph lessens the impact of the italics. I’d probably
• A nitpick: I mistakenly used hyphens in place of en- or em-dashes several times in the paper. (Adam Gleave told me about this mistake in 2019).
• It’s not obvious from the paper that the policies learned by the RNNs are actually optimal – I wish I included more of the results where I verified the optimality of the policies with classic POMDP solvers.

Discuss

In this post, I will try and outline the arguments for stopping AI.1 I’m not going to argue for them in detail. I’m just gonna try and get the most important points out on paper. We can fill in the details later.

Why is AI so dangerous?

The first thing to understand is that AI is not chatbots. AI is a general-purpose technology that can be trained to do everything humans can, and more. Right now, AI is mostly used to control computers. But robots are getting better all the time. Robots are the general-purpose hardware, AI is the general-purpose software. Put them together and you get machines that can do all the things.

AI is also advancing extremely quickly, repeatedly exceeding expert’s expectations. AI is already super-human in many ways, but not all ways. By the time it is better than humans across the board, it will be vastly better in some ways. AI can already think much faster than a human and has much broader knowledge than any individual person.

What are the biggest risks?

What will happen when AI is super-human in every regard, including not just IQ, but street smarts, emotional intelligence, intuition, physical grace, social maneuvering and politics, charisma, and so on? We don’t know, but if we are going to become a “second-class species” that’s obviously incredibly concerning. AI might literally lead to human extinction, in the same way that humans have caused many other species to go extinct.

Experts such as myself are doing just about everything they can to sound the alarm about the risk of human extinction. It’s not just a general uneasiness about something becoming smarter than us. Today’s AI systems “go rogue” and disobey commands, and we don’t know how to make them not do that. Many who aren’t sounding the alarm are focused on researching this sort of problem; unfortunately, many of those people work at the AI companies and that’s part of why they’re not sounding the alarm, too.

If super-human AIs — especially robots — go rogue, we might not be able to stop them. We couldn’t just unplug them, like we can with computers. But again, even without controlling robots, AIs might manage to take over, e.g. by playing different humans against each other. And actually, the AI doesn’t even have to turn against us, or turn us against each other. Companies and countries (and individuals) are already against each other, and are giving AI more and more power, and sacrificing their values in the process, in order to “win”.

There are other massive risks that would be a big enough deal to warrant stopping AI, even if we ignore the whole risk of extinction thing, as people sometimes do. AI could take everyone’s jobs, and then we might struggle to get the basic resources we need to survive. AI could concentrate power in the hands of authoritarian leaders or AI companies. It could destroy democracy and our way of life.

You might object that we don’t know if any of these things are actually going to happen. That’s true, but that just means we shouldn’t risk it.

What’s the plan?

So, we need a plan for dealing with this. There are basically two kinds of plans: 1) Stay in control of AI2 as it becomes increasingly super-human and increasingly powerful, 2) Stop AI from getting too powerful in the first place. At the moment, there are no good plans of type (1), for staying in control. There are some OK ones, that might work, but nothing we can count on. So we need to stop AI, if we can. And we can, so we should.

Why don’t other plans work?

I’ve talked a bit about how we can stop AI previously. I’ll say more about why the other plans don’t work in a future post.

Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.

Share

1

i.e. instituting an indefinite global pause

2

If we’re going with this sort of plan, there’s the additional questions of who should be in control, and what that even means and so on (Am I in control if I just follow my AI advisor’s advice all the time?). But that’s out of scope for this post.

Discuss

It seems that AI safety is at least partly bottlenecked by a lack of orgs. To help address that, we’ve added a page to AISafety.com aimed at lowering the friction for starting one: AISafety.com/founders.

This page was built largely as the result of a suggestion from @Ryan Kidd, who found he was frequently sharing the same set of resources with potential founders and realised it would be useful to have something similar publicly available.

It lists:

• Fiscal sponsors
• Incubators
• VCs
• Articles and tools

As with all resources on AISafety.com, we put substantial bandwidth into making sure the information on this page is accurate and up to date. If you have any feedback, please let us know in the comments or via the Suggest buttons on the page.

This is the 11th resource page on the site. Here's the full list:

• courses for self-study
• communities, both local and online
• upcoming events and training programs (plus a weekly newsletter)
• funders offering support for AI safety projects (and a newsletter announcing new opportunities)
• field map displaying the key orgs, programs, and projects in AI safety
• advisors offering guidance calls to help individuals most effectively contribute to AI safety
• media channels (newsletters, podcasts, etc.) for staying informed
• field-building projects seeking volunteer support
• jobs (a filtered subset of 80k’s database)
• a guide for donating to the field
• founder toolkit for starting and growing an AI safety org

Discuss

Epistemic status: Completely schizo galaxy-brained theory

Lightcone[1] operates on a "generalist" philosophy. Most of our full-time staff have the title "generalist", and in any given year they work on a wide variety of tasks — from software development on the LessWrong codebase to fixing an overflowing toilet at Lighthaven, our 30,000 sq. ft. campus.

One of our core rules is that you should not delegate a task you don't know how to perform yourself. This is a very intense rule and has lots of implications about how we operate, so I've spent a lot of time watching people learn things they didn't previously know how to do.

My overall observation (and why we have the rule) is that smart people can learn almost anything. Across a wide range of tasks, most of the variance in performance is explained by general intelligence (foremost) and conscientiousness (secondmost), not expertise. Of course, if you compare yourself to someone who's done a task thousands of times you'll lag behind for a while — but people plateau surprisingly quickly. Having worked with experts across many industries, and having dabbled in the literature around skill transfer and training, there seems to be little difference within an industry between someone four years in and someone twenty years in, once you control for intelligence and conscientiousness.

But sometimes someone on my team does actually truly struggle to get better at a task, even if they are smart. Or I notice that if I were to try to get them to do something, they would have no idea how to even get started unless they spent at the very least multiple months, if not multiple years, acquiring the foundations necessary to do so.

So the question becomes: What determines whether someone is capable of relatively quickly acquiring expert-level performance in domains ranging from preparing a legal defense, to preparing an architectural plan, to physically renovating a bathroom, to programming a conference schedule app?

And my current, schizo galaxy-brained theory is that there are exactly 4 skills:

1. Design skills: The ability to make good frontend design decisions, writing and explaining yourself well, designing a room, writing a good legal defense, knowing how to architect a complicated software system
2. Technical skills: Follow and perform mathematical proofs, know how to program, make Fermi estimates, make solid analytic arguments, read and understand a paper in STEM, follow economic arguments, make a business plan, perform structural calculations for your architectural plans
3. Management skills: Know how to hire people, know how to give employees feedback, generally manage people, navigate difficult organizational politics
4. Physical skills: Be expert level at any sport, have the physical dexterity to renovate a room by yourself, know how to dance

If you are good at any task in any of those categories, you can become expert-level within 6 months at any other task in the same category.

Now why these exact 4 skills?

IDK, it kind of fits the data I've observed. But here is roughly how I came to believe what I believe:

First: across all tasks, performance correlates highly with general intelligence, and this dominates everything else. But clearly there's non-trivial variance left after controlling for it.

Then, there's an obvious divide between STEM and the humanities. Ask someone with a legal, history, or non-analytic-philosophy background to learn programming and mostly they bounce off or expect a multi-year training journey. Ask someone with a STEM degree to learn programming and it goes pretty well even if they've never programmed before.

Similarly, when I talk to people with a legal or humanities background and ask them about complicated frontend design decisions, they usually give surprisingly good input! They will pretty quickly jump into the fray of trying to model the user, figure out what a good product or information ontology, and have a sense of style about its presentation.

So that's it. There are exactly two skills. "Technical skills" and "Design Skills".

Then I tried to manage people. That... didn't go so well. Not only that, when I tried to get people on my team to manage other people, they also sucked at it!

So I learned that if I want to predict who will be good at management, I need to pay attention to whether they've managed other people before, and expect many months of practice until they are decent at it. Maybe it's a completely new cognitive domain, maybe it's just a domain where skill transfer is very hard and feedback loops are very slow and so it just takes everyone a while to learn the basic lessons, but nevertheless, if I want to predict performance at Lightcone, I gotta model people's management skills separately.

And then I tried to renovate a hotel.

And while the people on my team really ended up surprisingly good at a very wide range of tasks associated with construction and construction management, it also became clear that no one on my team would be able to perform the actual labor that our general contractors were able to perform. And also that they would totally smoke us in any sports competition. And that if I wanted to get someone on my team involved in the daily construction work, I sure expect that they would need many months of getting into shape and developing the right kind of physical skills.

So 4 skills it is.

Now, am I confident I have seen all skills there are in the world, such that no additional cluster will arise? Actually, yeah, kind of.

I have been walking through the world trying to keep track of what kind of career many of my acquaintances and colleagues go into for something like the last 2-3 years, and haven't really noticed any big holes. I have also been actively trying to think about careers that currently seem off limits to someone who has basic expertise in these 4 skill domains, and I have so far not been able to find something. My guess is if there is something I am missing it will be in something less career oriented.[2]

Need someone to build a script that automates filling out some business forms?
Give your econ masters student 3 months to learn programming and he can do it.

Need someone to drive your marketing push?
Give your interior designer 2 months to figure it out.

Need someone to head your internal legal department, double check the work of your lawyers, and prepare your legal defense in a high stakes trial?
Give your very smart frontend designer 3 months and they will go toe-to-toe with your lawyers.

Want to promote an engineer who has never managed anyone before to a manager?
Well, you better strap in for a year or more of pain while they acquire this completely new skill domain and traumatize all your new interns while doing so.

Want to get your backend engineer who is not good at writing, and is not good at interior design, to start taking more charge over your frontend?
Expect them to suck for at least a year until they can start competing with the smart designers on your team.

Want to get your quant finance guy who has never worked on a big codebase to start writing maintainable code and make nice clean Pull Requests?
Well tough luck, predict many months of telling them that yes, it is actually important that anyone can read your code and figure out how to modify these abstractions you've created.

Want to get your philosophy grad student dropout who has never done physical labor in his life to start managing construction projects and get their hands dirty?
Expect at least a year of getting into shape and used to the work, if they don't bounce off completely (though many subtasks of construction can be done with pretty little physical alacrity).

Give it a try yourself!

(Unhappy with any of my classifications? Fight me in the comments!)

Is there any externally validated or scientific basis for any of this?

Yes! It's not like, total consensus in the field of psychometrics, but task performance being extremely g-loaded across a wide variety of tasks is very well supported. People can really learn a very wide range of skills if they are smart.

And then within intelligence, math tilt and verbal tilt tend to be commonly used abstractions in psychometric testing that are predictive of success in careers in STEM or humanities.[3] Math fits nicely onto the technical domain. Verbal fits nicely onto the design domain.

A generalized "physical skill" factor is also well-supported. First, enough high profile athletes have switched from being world class in one sport to being world class in another sport such that there must be substantial skill transfer for these domains to explain that outlier success.[4] Second, somewhat unsurprisingly, if you measure people's sports skill you will find a strong "General Motor Ability" factor that explains performance across a wide range of motor skills.[5]

On management? IDK, that one I haven't seen much support for, but it sure matches my experience. There is an emotional intelligence literature, but that construct adds extremely little on top of just general intelligence. My guess is it's just a task that's very important and has terrible feedback loops, so everyone needs to fail for a while before they get good at it, but who knows.

Design. Technical. Management. Physical skills. Long ago, the four nations lived together in harmony. Then, everything changed when the Management Nation attacked. Only the True Generalist, master of all four elements, could stop them, but when the world needed him most, he vanished.

1. ^

   the organization I run, and which runs the website you're reading

2. ^

   If there is a missing cluster, I can imagine it being some more "relational" skillset around doing high-quality emotional labor, or maybe something genuinely associated with age and wisdom where certain skill domains are just really hard to perform well in without being at least 35+ and having the associated life experience. But I don't currently think such a cluster exists, and that four is really the right number.

3. ^

   https://www.sciencedirect.com/science/article/abs/pii/S0191886924004860

4. ^

   Claude lists: "Bo Jackson (NFL All-Pro + MLB All-Star). Deion Sanders (NFL HOF + MLB). Rebecca Romero (Olympic silver in rowing, then Olympic gold in cycling four years later — different disciplines entirely). Clara Hughes (Olympic medals in both speed skating and cycling). Rugby → NFL is a well-trodden path (Jarryd Hayne, Christian Wade)"

5. ^

   This research is a bit more controversial than I expected, but I don't really understand the controversy. There are definitely some people in the field who insist on there not being a strong general motor ability factor. IMO this study also points in the direction of there being a general motor ability.

Discuss

I was so excited about the first BIDA dance that I arrived two weeks early. I biked over from Medford to the Park Av Church in Arlington and was really disappointed to find the hall was empty. But I came back when the dance was actually happening, and it was fantastic.

It immediately became my favorite dance. I started volunteering, first out of frugality (volunteers get in free!) and then out of a sense of wanting to contribute, and in 2010 I joined the board. Over the past 16 years I've done just about everything at some point except treasurer, and now I'm stepping away.

It's not that I think BIDA is doing something wrong; quite the opposite! We're seeing record attendance, finances are good, so many fun dancers, and many people who want to pitch in. I noticed I would have been the seventh person running for three board spots, and realized it was a good time to let someone else have a turn. I'm excited to see what Emma, Harris, Bret, Veer, Casey, Naomi, Clara, and Persis do!

This seems like a good time to look back over how BIDA and the Boston dance community have changed over my time organizing.

The biggest change is that BIDA is now Boston's main contra dance. This is kind of hard for me to believe, since we spent so many years as a small dance that tried to fill niches that were not well covered by the many other area dances. We've gone from essentially not booking established bands to booking them regularly, and with our attendance-based bonuses are one of the best-paying dances in the country. I do really enjoy the higher level of musicianship now, but am also really glad Boston Open Contras exists (along with BIDA's open bands and family dance bands) to provide a lower-stakes environment.

The next largest change is probably the switch to gender-free calling (more history), and the level of role freedom that has come along with that. In 2010, I (and many others) would happily dance both roles, but if I was dancing the 'lady' role I had to be 100% on it because if anything went wrong it was my fault. Beginners were strongly discouraged from dancing 'switch', which also discouraged same-gender couples. And while this never happened to me in Boston, conservative men elsewhere would occasionally refuse any sort of physical contact if I encountered them in line while dancing 'lady'. When I look at the dancers now, it's amazing how people have really taken up this freedom to dance any role with any partner, which I feel really good about.

Some smaller changes:

• BIDA went from 1x/month to 3x/month, most recently by adding a monthly afternoon dance. Since we take the hottest part of summer off, this means going from ~10 to ~28 dances annually.

• We now have a dance weekend, Beantown Stomp. I kicked this off in May 2018, we had our first one in March 2019 and it's now an established and anticipated event that people fly to from across the country. I'm especially grateful for Naomi for taking the lead for 2023 (and beyond!) when I was too burnt out on organizing cancelled events (2020, 2021).

• We have occasional family dances and livetronica (Spark in the Dark) events.

• Our events are still intergenerational, but differently so. In 2010 most dancers were baby boomers; while BIDA was unusual in how many millennials we had, we were still 50%+ baby boomers. At this point I'd guess our dances are fewer than 10% baby boomers: many have aged out of dancing, and many millennial-and-younger dancers have joined. This is also reflected in the board's focus: the initial board was primarily mid-20s people thinking about how to get more 15-35yos dancing, but since we've succeeded at this it's no longer a focus.

• We now schedule (and pay) hall managers. In 2010 we just expected most board members would be at most dances and this would give us enough coverage.

• BIDA is a lot more organizationally mature. Minutes from the early days say things like "We agreed not to have a President. Instead, we'll use everyone in the board to make sure that we stay on top of things." This turned out not to work very well, and instead specific roles are in charge of staying on top of specific things, with the intraboard coordinator handling things by default.

• We were still bouncing around between a few halls, and now we're always at the Cambridge Masonic Hall.

• We're a legal entity now, incorporated as a Massachusetts non-profit.

• We set up a safety policy, with a committee to handle issues as they come up.

• There used to be a lot more of a mentoring focus. Early dances were often two experienced musicians plus a new musician. Callers would typically have a shadow. Every dance allowed sit-ins (off mic, behind the band). We hosted jams about as often as dances. I see this change as pretty natural, and I think a lot of this is now happening informally outside of BIDA.

Organizing BIDA has been a big part of my identity, but I think it's healthy for the organization to have people cycle through, and I'm confident it's in good hands. Very excited to start attending dances just as a dancer, with no formal responsibility!

Comment via: facebook

Discuss

A number of years ago Numberphile published a video on the behavior of higher dimensional spheres with the title “Strange Spheres in Higher Dimensions”. I'd recommend one watches the video before reading this post, in that video Matt Parker presents a construct where spheres are placed inside a cube pressing against its faces, in the remaining central volume an additional sphere is set with a radius so that it kisses the “padding spheres”, this build is then scaled to higher dimensions where Parker observes that the central sphere increases in size constantly until its surface goes through the face of the cube, an apparently impossible configuration. He then jokingly concludes that the only way to make sense of it is to imagine higher dimensional spheres as “spiky”, so that the spikes can go through the face while the rest of the volume remains confined inside the hypercube.

Recently, for some reason that same video ended up in my feed and I ended up watching it again, I was left rather dissatisfied with the conclusion. I am not a mathematician but I was always fascinated with higher dimensions and when it comes to geometry, I often found that seeing is believing.

Plotting sections of the construct can help us make sense of the problem, we’ll start with the simple 2d version, here the central sphere has a radius of mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msqrt { display: inline-block; text-align: left; } mjx-root { display: inline-block; white-space: nowrap; } mjx-surd { display: inline-block; vertical-align: top; } mjx-sqrt { display: inline-block; padding-top: .07em; } mjx-sqrt > mjx-box { border-top: .07em solid; } mjx-sqrt.mjx-tall > mjx-box { padding-left: .3em; margin-left: -.3em; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mn { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c.mjx-c221A::before { padding: 0.8em 0.853em 0.2em 0; content: "\221A"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c1D45B.TEX-I::before { padding: 0.442em 0.6em 0.011em 0; content: "n"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } :

Fig. 1 not much to see here, everything is visible on the plane

The true solution of the “mystery” is understanding the diagonal of the cube in higher dimensions. The diagonal of the measure polytope in n dimensions is times the size, so it becomes larger and larger as we increase the number of dimensions. This is easy to understand: for a unit square the diagonal is , to calculate the size of the diagonal of a cube starting from this we must solve the hypotenuse of a right triangle where the catheti are and 1; the result, of course is . If we wanna go up another dimension this process must be repeated one more time, so the radicand must grow by one integer.

The diagonal of a face also grows in a similar fashion, since it only has one less dimension that the hypercube, for example in the 9th dimension the diagonal of a face of a unit hypercube is .

Understanding the 3d version of the problem is not too hard but to keep things simple I will show a 2d section only, to obtain this section we will slice the cube through the diagonal as shown in fig. 2.

Fig. 2 a render of the problem for n=3, the segmented line is the edge of the section

The highlighted section is shown in the next image, notice a gap appears in the center as not all of the padding spheres are touching each other. Also, the padding spheres appear to no longer be contacting all sides of the cube since some of the contact points do not lie in this section.

Fig. 3 The radius of the central sphere has grown, it is now

This method is quite useful, since the plane we chose passes through the centers of both the central and padding spheres, the sections of these objects will always appear as circles of radius 1 no matter how many dimensions are we working with. Now let us move to the 4th dimension:

Fig. 4 in 4d the central sphere has now the same radius of the padding spheres, 1

This pattern continues, as the diagonal of the face continues to grow so does the diameter of the central sphere, in 9 dimensions the sphere is now contacting the faces of the hypercube:

Fig. 5 The diagonal of a face is now almost 3 times the side

Due to symmetry the radius of the central sphere is always equal to the distance between the surface of a padding sphere and the closest corner. In 10 dimensions part of the central sphere is now escaping the inner volume entirely:

Fig. 6 The radius of the inner sphere is now , or about 2.16

This pattern continues as n increases and more and more of the central sphere escapes the polytope. Higher dimensional spheres are not spiky but the faces of higher dimensional cubes have very large diagonals compared to the side through which the central sphere can escape.

Discuss

In my previous post I found that activation steering worked with KV_cache and not with hidden state steering.

So I decided to look at the PCA with methods such as logit lens and activation steering

Quick Summary:

• PC1 from hidden state activations strongly seems to correlate with the <|eocot|> or end of chain of thought token across all latent positions for the GSM8K dataset
• Added Critiques of CodI near the end

Experimental setupCoDI model

I use the publicly available CODI Llama 3.2 1B checkpoint from Can we interpret latent reasoning using current mechanistic interpretability tools? 

Tuned Logit LensTo create my tuned logit lens implementation I used the code implementation for the training of Tuned logit lens from Eliciting Latent Predictions from Transformers with the Tuned Lens

Activation Steering

1. Embedding steering

Getting the average hidden state from each latent vector and using the difference between latent vector A and B to steer the hidden states.

 Since codi uses the kv values on eot token. To get new kv values that contain the info from the steered vector I needed to steer latent 1 -> run codi for one additional latent and then get the kv values of latent 2 and see the output.

2. KV cache Steering

Steering the KV cache and adding the steered KV cache directly onto the codi model. Directly adding average difference in kv values to past_key_values.

ExperimentsPCA Logit Lens

• With the hidden states with PCA direction 1 there seems to be a clear <|eocot|> direction in the CoDI model which is interesting as the <|eocot|> token is steered with the kv cache to output the response for latent reasoning.
• Looking at the PCA for KV cache they do not look like there is interpretable directions all of the directions seem to have similar variances

PCA Hidden State Activations

• When looking at the PCA activations for the hidden states there seems to be a clear diagonal separation for even and odd steps for PCA 1 and PCA2 for hidden state activations during latent reasoning. This matches up with previous post results where there was a clear distinction between even and odd latents supporting the conclusion from  the Scratchpad Thinking paper.

PCA KV cache

• The PCA directions do not seem to have the different latent steps cluster like with hidden state and looks random

PCA Steering

• PCA steering seems relatively uniform for activation and kv value PCA directions
• Activation Steering with hidden state requires a forward pass due to the fact CoDI only keeps the kv values. So to make a fair comparison of PCA steering I ran kv steering through a forward pass and found KV steering matched a random vector after forward pass.

Critique of CoDI

Note: this section is opinionated. The claims below are my interpretation and speculation, not established findings — treat them as hypotheses worth testing rather than conclusions.

The following critique draws on findings from this sprint's PCA analysis alongside Can we interpret latent reasoning using current mechanistic interpretability tools?, the Scratchpad Thinking paper, and my previous lesswrong post sprints 

• CoDI works by running through n latent forward passes then only keeps the kv_cache which is used to steer the <|eocot|> token CoDI removes the hidden state computed when outputting the answer with latent reasoning. CoDI acts like a goldfish in the sense that after latent reasoning it forgets what happens during latent reasoning which might make CoDI not scale. After 6 latent reasoning steps and outputting a token the kv values are not saved for the generation of future tokens.
• This makes it not possible to steer codi traditionally with hidden states as for the <|eocot|> token only the kv values are kept. This makes hidden state steering require another latent reasoning pass to get updated kv values to use to generate the answer.
• The forward pass step for hidden state activation steering explains why for the later layers the random vector did not change the accuracy for CoDI.
• KV values can be steered without an additional forward pass so they can be used to meaningfully change the accuracy of the model to be better after steering. However, after the forward pass the performance more closely matches the Random vector.
• With how the latent forward passes have all their data stored inside the kv cache the limitations of how many steps that can be done with latent reasoning might be a limitation of the kv cache.
• The kv cache might be similar to how tokenizing the step at each step forces the hidden state to be one token. But, instead it is forcing n steps to store their information on the kv values. This could explain why after latent step 5 the accuracy seems to start decreasing as the kv cache is saturated and can’t store more info
• Due to how kv cache does latent reasoning is not saved unlike normal cot which has the previous tokens listed out so the model in future tokens is able to refer to past tokens.

Future Work

• Create a different version of latent reasoning that is not CoDI
• Attempt to create a CoDI variant that can take in hidden state values so a lot of the information is simply not lost during the latent reasoning generation.
• Look into other latent reasoning models

Discuss

There’s a genre of book that’s perennially popular. Some examples include:

• 7 Habits of Highly Effective People

• Getting Things Done

• How to Win Friends and Influence People

• I’m Ok, You’re Ok

What these books have in common, aside from being self-help, is that they’re attempts to help people make the transition from the pre-rational, pre-systematic thought most of us have entering adulthood to the rational, systematic, modern, and self-authoring thought of Kegan Stage 4.

This process is often plagued with difficulty, as Kegan himself explores in In Over Our Heads, and is especially difficult for people who master Stage 4 thinking in one area of their lives but struggle with it in others. Folks like doctors, lawyers, scientists, and engineers are masters of Stage 4 thinking by the time they graduate from college, but usually only within their domain of study. They can easily spend decades with their personal and social thinking trailing in Stage 2 or 3, and suffer all the more for it because they know more would be possible if they could just figure out how things work.

Thus, I was pleased to recently come across a copy of The Unwritten Laws of Engineering. Originally published as a series of three articles by W. J. King in Mechanical Engineering magazine in 1944, the book I found was a second edition with revisions and additions by James G. Skakoon. And although the original advice is now several decades old, it still reads well for professionals learning to operate at Stage 4 in their work lives.

The book is relatively short at just 60 pages, but in that space its direct, to-the-point style does a good job of explaining what should be obvious. Some of its advice includes such nuggets as:

• Confirm your instructions and the other person’s commitments in writing.

• Return your messages.

• Whatever your supervisor wants done deserves top priority.

• Meetings should be neither too large nor too small.

• Cultivate the habit of making brisk, clean-cut decisions.

• Regard your personal integrity as one of your most important assets.

• Beware of what you commit to writing and who will read it.

It ends with a brief discussion of how to analyze yourself, as a system, just like an engineer would analyze their work. It encourages readers to find their strengths and learn to exploit them, and also to notice that one’s passions and desires may not always lead to the happiest and best life.

I get the impression this book is a popular graduation gift. Maybe I would have seen a copy earlier if, despite the various job titles I’ve held, I’d been an engineer instead of a programmer. But I wish I had, both for myself and for the engineers I’ve managed who, despite years of experience, held themselves back by not applying the same systematic approach to themselves that they applied to their work, for it was only learning to treat myself as a system that began to learn how to take control of my own life.

That said, like most books in this genre, I’m sure it’s lost on the people who need to read it when they first do. The lessons one must learn to make the Stage 4 transition are complex, have to be lived, and can’t be picked up in an afternoon from a book, but the books do help! They plant the seeds of ideas in the minds of their readers, and as best I can tell, The Unwritten Laws of Engineering is as good at sowing as any book in the genre.

Discuss

[Author's note: this post is the short version meant to provide quick, actionable advice for treating OCD. The narrative version is longer but better, in my opinion.]

Obsessive Compulsive Disorder (OCD)

OCD is an anxiety disorder that feeds on uncertainty. It affects 1% of people worldwide, and the World Health Organization ranks OCD among the top 10 most disabling illnesses. 

Everyone feels uncertain from time to time, but people with OCD struggle to accept any amount of uncertainty for specific fears they have.

As an example, normal people wash their hands only once and feel a sense of satisfaction that they’ve effectively cleaned the germs off their hands. Someone with OCD, however, may not feel that sense of satisfaction with washing their hands just once, or even twice in a row. What if only 99.99% of germs are killed? And what if the leftover 0.01% of germs infects them and poisons them and rots their organs away and…and…OH GOD, SPREAD TO OTHER PEOPLE AND KILL THEIR FAMILY AND START THE NEXT GLOBAL PANDEMIC!!! So to avoid this terrible thought, someone with OCD might wash their hands 23 times in a row—to the point of rubbing their skin so much that it bleeds—in order to achieve that satisfied feeling of being clean.

This is an example of what’s known as Contamination OCD, a common subtype of the disorder. While I’ve never been a compulsive hand washer, at 7 years old I was mockingly referred to as “the shoe nazi” by my family. Anytime someone wore shoes in the house, I felt compelled to sweep everywhere they had walked. Because my family wasn’t, and still isn’t, educated on OCD, they just thought it was a weird childish quirk. Little did they know that there was a monster inside of me ordering me to clean because dirt felt dangerous. So when my mom decided to break me of my cleaning compulsion, by putting shoes on my feet and dragging my kicking legs across the floor while I screamed my head off, she didn’t realize the extent of the sheer terror she was actually causing me in my brain. Although aggressive, her tactic worked. I learned that shoes in the house don’t lead to my demise and I began tolerating them inside.

To avoid the feeling of life-ending panic at the core of OCD, people with the disorder adopt compulsive behaviors that give them an illusion of control and temporarily dampen their anxiety. This process is excellently described in psychologist George Weinberg’s 1993 book Invisible Masters: Compulsions and the Fear that Drives Them.

Every compulsion is an act of terror. It is an attempt to regulate something concrete and controllable because the person cannot identify and control some real psychological problem. The victim of a compulsion is performing a symbolic ritual as a way of subduing ideas or feelings that seem too hideous to be accepted…that feel too utterly out of control.

Because the problem is deep-rooted, a person’s engaging in any compulsive activity can bring at most momentary peace. The real underlying problem remains, and before long the person feels the need to engage in the compulsive behavior again.

The underlying dread and annihilatory sense of panic is so utterly disturbing, that the OCD sufferer’s mind will create rationalizations in order to avoid that pain. As Weinberg explains it:

Some obsessions appear so sensible that the sufferer feels delinquent not paying attention to them. The mind offers up the false logic that the cost of doing the thing, even if it was just done, is trivial, whereas the downside of not doing it may be quite serious, possibly disastrous. And so the compulsive engages in the ritual, just to be on the safe side.

Interestingly, the person with OCD doesn’t need to have obsessive thoughts or rationalizations for them to still be afflicted by the dread-to-compulsion mini-loop.

The only things required to keep this mini-loop-from-hell alive is the feeling of dread, and to perform a compulsion that legitimizes and reinforces the underlying dread. The obsessive thoughts or rationalizations are optional, and often show up when one attempts to resist a compulsion.

Obsessional thoughts merely belong to the urge, which is generated from deep inside. They are excuses for the impulse to engage in the compulsive behavior. The person would feel the same compulsive urge without any thoughts at all.

And this tracked with my cleaning experience. I never had any clear thoughts as for why I had to sweep everywhere, I just knew that I had to. And when my mom dragged me across the floor with my shoes on, that’s when the obsessive thoughts and rationalizations (ie: my brain screaming that I’m going to die) arrived to persuade me to engage in my compulsive behavior.

Deconstructing OCD into its two parts

People with OCD have: (1) severe anxiety, and (2) disordered thoughts. 

(1) Severe Anxiety

Everyone has an internal panic response. If you’re driving, and suddenly a car is hurtling straight at you and you’re on the verge of colliding, you panic and veer away! Thank god for your ability to panic, because that heightened state of alertness saved your life.

The best way I can describe OCD is that my internal panic alarm (which again is a normal part of the human experience) misfires at inappropriate times. I have an exaggerated sense of what could end my life, and perform compulsions to regulate my panic. This is why so many people with OCD go without treatment for years—we’re just obediently following our bodies’ panic system, like everyone else does.[1]

(2) Disordered Thoughts

By chance, I stumbled upon this blog post in January and it completely changed my life. In it, the author explains Metacognitive Therapy (MCT). 

The process of metacognition is to observe and evaluate your thoughts as if you are an outside observer. This is helpful for anyone with anxiety or depression because not all thoughts should be taken seriously.

MCT proposes that OCD thoughts manifest in three ways: the past, the present, and the future.

1. In the past, the OCD brain endlessly ruminates about what I could have done better, to the point of mentally beating myself up.
2. In the present, the OCD brain monitors for threats that could end my life, and produces a strong feeling of panic to compel me to change course or to perform a ritualistic compulsion to temporarily alleviate my life-ending fear.
3. In the future, the OCD brain worries about potential threats that could end my life if I'm not careful and plan accordingly. It typically manifests for me in a checking compulsion where I constantly Google variations of the same thing over and over.

Funnily enough while writing this, I realized that my blog post last December titled “Your Digital Footprint Could Make You Unemployable” was actually a manifestation of my future worry OCD. Oops! While the logic of the post is somewhat sound, what you didn't see (because I didn't write about it) was the two weeks I spent compulsively Googling myself, trying to quell the annihilatory panic I wound myself into. I was worried that employers will not approve of my public blog presence, and that I'll end up blacklisted and unemployed forever, destitute and forced to live on the streets, and then I'll get AIDS and die. That's usually how a lot of my disordered thinking goes: all roads lead to becoming destitute on the street, getting AIDS, and dying. Fun!

So what caused me to finally get help with my OCD?

Throughout 2025, my brain was slowly trying to convince me that I might be developing a fatal skin disease. Did I actually have one? No. But my brain was convinced. So to quell my fear, I would compulsively check all over my body to look for symptoms. By November of last year, just 6 months ago, I was spending 3 to 4 hours each day performing my checking compulsion. But as I did the compulsion more and more, the relief from my fear became more and more temporary. I thought I was losing my mind.

I’ve always kind of known that I might have OCD (it’s ~40% heritable and my grandfather had it), so I sought out an OCD specialist. I winced at the price—$200/hour—but ultimately gave it a try because I was starting to not be able to function at work.

Agreeing to see a therapist made me feel pathetic (you know, cuz classic male conditioning tells us that asking for help makes us weak). Additionally, I felt apprehensive that all this stuff was just in my head and that I don't actually have OCD (which I later learned is a classic tactic by OCD). After neurotically ranting at my therapist for 20 minutes on all the reasons why I might have OCD (because I wanted to justify my being there), she sympathetically looked at me and smiled, “Yes honey, sounds like OCD.”

That validation was the best thing I got out of my two visits with her. I felt seen. I didn't feel irrationally crazy anymore. And I got help. I learned one basic technique to manage my skin disease checking compulsion[2], and after a few weeks the fear went away!

Solutions

As mentioned previously, OCD is primarily a mismanagement of the feeling of severe panic, and secondarily reinforced with disordered thoughts that are attempting to be helpful, but are unfortunately producing avoidant/compulsive behaviors and thoughts that legitimize the panic. I found three therapeutic practices to be effective.

1. Acceptance and Commitment Therapy (ACT) — Feelings are not problems that need to be solved. Feelings just want to be felt. So if we compulsively avoid negative feelings, like anxiety or panic, we unintentionally make them stronger. (Kind of like trying really hard to not think about a pink elephant—it'll only make you think of it more). But when we relax and sit in uncomfortable feelings, they get felt, processed, and then usually go away.
2. Exposure and Response Prevention (ERP) — If we voluntarily expose ourselves to experiences that give us anxiety, and we prevent a response by refusing to perform avoidant behaviors, we train ourselves to tolerate the feeling of anxiety. We learn that while the feeling of anxiety (or even panic) is unpleasant, it's not going to kill us.
3. Metacognitive Therapy (MCT) — Just like feelings, thoughts also like to be acknowledged and not suppressed. With OCD, ruminative thoughts, threat monitoring, and worrying about the future are unhelpfully trying to “solve” for unsolvable things in life. MCT suggests we observe these thoughts and label them non-judgmentally when they happen.

I combined all three of these into something I call “Panic Meditation”. Strap in, cuz this is about to get crazy.

The first week of February, I laid down on my bed and prepared to confront one of my fears that my OCD brain said would kill me: eye contact with beautiful women. (OCD preys on your innermost fears—mine was that women could potentially see me as a creep). 

First, I told myself that whatever negative feelings come up are okay to feel (ACT). Second, that any thoughts that arose to rationalize taking me out of the experience were part of my disordered thinking to try to keep me safe (MCT). With those two things in mind, I voluntarily exposed myself (ERP) to something that normally causes me terror: I imagined myself sustaining eye contact with a beautiful woman.

Panic Meditation

Immediately 8-out-of-10 pain shot across my body as my internal alarm system began blaring, “CODE RED. CODE RED.” My body started seizing and I gripped onto my sheets for dear life, reminding myself that I'm safe on my bed, meanwhile my brain screamed, “YOU’RE GOING TO DIE MOTHERFUCKER!!!”

Rationally, I knew that eye contact with a woman was not a life-threatening experience, yet emotionally my brain and body were signaling otherwise. 

After a minute of excruciating pain, my brain recognized that I was refusing to respond to the alarm system, so it decided to send in the cavalry: disordered thoughts.

My brain: “Dude, bail, you’re in a ton of pain right now.”

Me: “No,” I grimaced, “I’ll be okay if I stay in this experience.”

My brain: “No you won't, you're going to die. I've been protecting you your whole life, and you've listened to me your whole life, and guess what? You've never died. I'm trying to help you. When I send you the panic signal, you listen—that's our deal.”

Me: “Eye contact isn't going to kill me. Normal people do it all the time.”

This is where I messed up with my initial attempt at Panic Meditation. Disordered OCD thoughts shouldn't be taken seriously or debated. They'll win every time. So I lost the edge next.

My brain: “OH YES EYE CONTACT CAN KILL YOU! All you have to do is think about it for just half a second and you'll recognize that. But you're too stupid to do that on your own, so it's a good thing that I'm here. Put simply: with eye contact, a woman could intuit that you're attracted to her, but you're a terrible person that nobody could ever love, so it's only a matter of time until she feels creeped out by you, and then you're in BIG trouble, because she could start a rumor that ruins your reputation.”

Me: “So I can't even look at a woman?”

My brain: “Correctamundo, amigo! With your reputation ruined in your community, somebody might try to cancel you online and you'll be the new face of #MeToo. And, of course, at this point you'll get fired from your job. Need I remind you what will inevitably happen next?”

Me: “Jesus, what?”

My brain: “Branded as a sex offender, you’ll never be able to get a job again, you'll soon run out of money, get kicked out of your apartment, end up on the street, get AIDS and then die. Case closed.”

At this point, my overall pain was a 9-out-of-10. Almost unbearable. My body's alarm system was blaring. My brain was egging me on to quit and save myself while I still could. I had sweat a puddle on my bed. 

Only 5 minutes had passed.

Gripping my bed sheets, I remembered what MCT taught me, and I whispered aloud the softest, “no…”

My brain, taking a victory lap, scoffed, “What was that, champ?”

Me, more firmly: “No.”

My brain: “Excuse me??”

Me: “No, you’re wrong.”

My brain: “Ho ho! You've never bested me before,” cracks knuckles, “but you can try—let's dance!”

Me, weakly: “No, I'm not going to debate you. But I do have some magic spells I'm going to cast on you.”

My brain: “What a fucking nerd you are. Fine, taste this Hellfire! YOU'RE A FAILURE AND YOU MAKE EVERY WOMAN AROUND YOU UNCOMFORTABLE!”

Me: “I acknowledge that thought, but I’m going to label you as one of the three MCT categories: Threat Monitoring!”

My brain: “... … …pffft, you think that scares me? That’s nothing. Here’s another variation. WOMEN HATE YOU AND YOU SHOULD BE ASHAMED TO EVEN LOOK AT THEM!”

Me, more forcefully: “I acknowledge that thought, but still—threat monitoring.”

My brain: “... … …yeah, well, of course I’m monitoring for threats. That’s my job. Look dude, we already covered this. You’ll end up destitute with AIDS and die!”

Me: “I acknowledge that thought, but you’ve switched and are now doing the second MCT category. I label you: Worrying!”

My brain, sputtering: “You fucking idiot and loser. Need I remind you of the past? Every woman you’ve ever liked hasn’t reciprocated your interest, at least not for long. Let’s quit this silly business and go analyze every past mistake you’ve ever made with a woman, for hours and hours, like we usually do. It’s one of your favorite hobbies. Plus, who knows, maybe you’ll finally uncover the real reason you failed so that you can be certain to not fuck up again.”

Me: “I acknowledge that thought, but you’ve again switched to the third MCT category. I label you: Ruminating!”

My brain: “Oh you’ve done it now, bucko. I was trying to protect you, but now I’m going to make your life a living hell!”

We clashed for the next 10 minutes. My brain sent me increasingly more convincing rationalizations, each one more subtle than the last in an effort to get me to bail from the experience. But each time I caught on and started responding with the one-word labels of “ruminating”, “threat monitoring”, or “worrying”. Additionally, my brain sent me other random thoughts, like song lyrics or what I planned to eat for lunch, but I ignored those, too, and kept returning to imagining making eye contact with a woman, which kept my anxiety level high. I wanted to stay in the feeling of panic and not allow myself to escape it.

Each time I started clenching my sheets and my whole body seized with pain, I would deliberately practice unclenching and relaxing—ERP is about preventing a stress response to a stimulus. ACT reminded me that this feeling is part of who I am and it’s not something I have to avoid. And MCT was there to help me label my disordered thoughts. After those 10 minutes, both me and my brain were exhausted.

My brain: “... … …fuck you.”

Me: “I acknowledge that thought and—”

My brain, interrupting: “—oh put a sock in it. I’m done here…for now. But enjoy this parting gift.”

Me: “What gift?”

My internal alarm system and pain shot through the fucking roof. I hung on for dear life, as if I was riding a roller coaster with tons of bolts missing. I kept reminding myself, “I’m not in danger, I’m safe on my bed. I’m not in danger, I’m safe on my bed. I’m not in danger…”

I rode out the feeling of annihilation for another 5 minutes, then finally got up. My first ever session of Panic Meditation lasted a total of 20 minutes. I took a shower to get the sweat off.[3]

After my shower, I remembered there was a run club meetup that night. When I showed up, I accidentally made eye contact with this cute woman I normally ignore because looking at her causes a micro-panic sensation in me (like I’m getting punched in the gut and having the wind knocked out of me). But this time…I didn’t feel panic! I let this register for a moment. My god, I don’t feel panic right now! Ha hah!! OH MY GOD!!! THIS IS SO WEIRD! And then it got better.

I started talking to her. I made her laugh. The rest of the night I talked to a bunch of new people and didn’t feel any social anxiety. My god…what have I discovered?

Over the course of just one week of daily 20 minute Panic Meditation sessions, my feeling of life-ending panic shriveled up because I repeatedly refused to react to it. By week two, I could no longer induce the feeling of annihilatory panic in myself, and instead could only feel a mild (yet still unpleasant) feeling of anxiety.

From talking to a random stranger at a coffee shop weeks later (who turned out to be a psychologist), I learned that what I invented with Panic Meditation is actually already an established psychotherapeutic technique called “flooding”. With flooding, patients voluntarily trigger incredibly strong/stressful emotions, and then try not to react to the experience so that they get used to it and not run away. This psychologist said that patients seldom agree to try flooding because it’s incredibly painful. It's kind of like ripping off a Band-Aid—quick short term pain, versus the prolonged pain of exposure therapy over many weeks, months, or years. He was astonished that (a) I independently created flooding without knowing about it, and (b) that I stuck with it despite the pain I experienced. Him saying that made me feel proud of myself.

Where I’m at today

A couple months later, sometimes intrusive thoughts arise, I gently label them, and they drift away. I’ve found that regular meditation 10 minutes per day is super helpful for decreasing the frequency of disordered thoughts. That, and regular exercise. They’re both probably working by lowering my overall stress levels.

After just two weeks of daily Panic Meditation at the start of February, I rarely experience annihilatory panic now. My nervous system has reset and is better calibrated to reality. When I do occasionally feel that I’m on the verge of imminent death, I don't allow that feeling to bother me. 

I still do the occasional Panic Meditation session when I need to process something that makes me feel nervous, or if I want to pre-experience the feeling of anxiety before a social event. But it’s never as intense as that first session I did.

I still experience general anxiety, but it’s likely the normal variety that we all sometimes have. I’m no longer disturbed by the feeling of anxiety or allow it to bully me. In fact, I get a little excited when I feel anxious because that usually means I’m on the edge of my social skills—a chance to level up if I push through!

Because most of my waking thoughts are no longer spent hating myself, or spiraling about imagined scenarios, I have way more time to write. Last month I wrote a guide on how to stop ruminating. Two weeks ago in another blog post, I wrote a section on how to stop threat monitoring with respect to social anxiety. 

With everything that OCD touched, my world got smaller: seemingly innocuous social interactions felt like life-or-death, minor decisions during my day became indictments of my moral character, spending money nearly induced panic-attacks (because I’ll soon become penniless, destitute on the streets, get AIDS, and die), touching “unclean” objects could give me an infectious disease, and flirting with women carried the weight of a potential prison sentence. Having OCD, I imagine, is like living in North Korea—any small misstep and you, and your entire family, could be sent off to prison for the rest of your lives, or executed.

I wrote this blog post because I don’t want my suffering to be in vain. I have three friends in my life who also have OCD, and many more friends that struggle with social anxiety (which I strongly suspect my techniques will equally work for treating; social anxiety seems to be a subset of OCD where a person obsesses about avoiding disapproval, and then compulsively restricts their behavior or speech to ensure that nobody dislikes them).

So to my fellow sufferers, I can say this. The good news is that OCD is incredibly predictable (ie: it will always try to escape uncertainty by producing anxiety, or generating unhelpful, overprotective thoughts), and therefore I’ve been able to treat it in such a short span of time (~95% of my symptoms resolved in just a couple weeks, and I continue to feel good several months later). So give Panic Meditation a try, if you want short term results. Or perhaps see an OCD therapist to ease you into treatment. Use ACT to embody and accept negative feelings, ERP to habituate to the feeling of panic while refusing to respond to it, and MCT to label disordered thoughts (and to not negotiate with terrorists). Hopefully you can escape like I did. Good luck.[4]

1. ^

   What first set me on the path to escaping my cycle of terror was psychologist Bruce Tift’s book Already Free: Buddhism Meets Psychotherapy on the Path of Liberation.

   In his book, Tift explains that many people incorrectly treat uncomfortable feelings, such as anxiety, like they’re problems that need to be fixed. Whereas Buddhism teaches that feelings cannot be solved or avoided. Feelings are simply signals from the body. Attempting to escape the feeling of anxiety by ignoring it, or using coping behaviors, only makes it worse in the long term.

   Some people go their whole lives avoiding the feeling of anxiety. But, paradoxically, the way to resolve anxiety, according to Tift, is to embrace it head-on.

   The daily end-of-the-world panic I feel is a part of me. That will never change. So I can either choose to avoid that terrifying feeling like I have my whole life (and use compulsions to temporarily alleviate my pain), or I can learn to have a relationship with my panic and accept that it's part of the uniqueness of who I am.

   What Tift is describing in Buddhism, has a western counterpart called Acceptance and Commitment Therapy (ACT). With ACT, you voluntarily accept any negative feelings that arise, and you don’t allow them to inhibit you from living your daily life. In doing this, those negative feelings never go away, but they lose their power over you. They can't control you anymore. 

   Let me repeat that: disturbing feelings will never go away. We can either accept them in all their ugliness, or forever go on coping for them with avoidant behavior.

   I can't do the latter. not anymore. After a lifetime of being a prisoner to my fear and compulsions, I’m dead tired from exhaustion. 

2. ^

   We each held onto the end of a rope, then we stood on opposite sides of the room. She told me to imagine that between us was this chasm that represented my greatest fear. She then started pulling on the rope. I pulled back to avoid falling in. She pulled harder. I pulled harder. After a minute of this, she asked, “Can you imagine another way to avoid falling in the chasm?” I thought about it, then dropped the rope. That was the point. Our greatest fears can only haunt us if we interact with them.

3. ^

   It’s important to note that, for the context of telling my story, I separated “me” and “my brain” into two different characters—it’s more entertaining that way. But inside my head there’s only one voice for my thoughts. So when I started experimenting with all this stuff, it was difficult to know which thoughts were OCD-thoughts and which ones were normal thoughts.

   In January of this year I joined a hot Pilates studio (because that would be a good place to be around lots of new people, especially women, and to practice preventing myself from doing OCD-based threat monitoring behaviors). Ironically, the one connection I made there was with a man, who turned out to be an OCD therapist! I asked him how people with OCD can distinguish between their thoughts. He gave me a simple heuristic: if you’re spending time wondering if a specific thought is OCD related, it probably is. I have found that to be true every time.

4. ^

   Further Resources:

   • OCD therapy
   • • Not all therapists are trained to treat OCD. You’ll want to find a specialist. Like I mentioned, mine charged me $200/hour. Your insurance might cover it, or not. Either way we’re talking about your health, which in my opinion, is worth spending money on. If you’re already informed on how OCD works (from reading this blog post and from consuming the following resources I’ve included below), then you’ll save time/money in therapy, and perhaps only need the therapist to guide you through some techniques for you to practice at home.
     • My friend recommended me Psychology Today which is how I found my OCD therapist.
     • There’s also NOCD, a reputable online OCD therapy service.
     • • Here’s a short film the folks at NOCD made showing the disordered thoughts and images of someone living with OCD.
     • Dr. K on YouTube is a mental health expert who makes excellent videos. Here’s his technical breakdown of OCD, statistics related to it, and why he’s an advocate of ERP for treating it.
   • Anti-anxiety medication
   • • Besides therapy, medication is the other gold standard for treating OCD. I didn’t mention it in this blog post because I don’t have any personal experience with it. I’ve heard from one of my friends, who has OCD, that anti-anxiety pills reduce their anxiety to a manageable level so that they can function in their daily life.
   • Brain Lock: Free Yourself from Obsessive-Compulsive Behavior by Jeffrey M. Schwartz, MD with Beverly Beyette
   • • I’m still in the middle of reading this book, but it’s good. The author is a psychiatrist at the UCLA School of Medicine and is a world-renowned expert on OCD. He’s done all kinds of studies showing that neurologically, the brains of people with OCD work differently than neurotypical brains. He's helped thousands of people with his recommended solutions for overcoming OCD. It might have been nice if I had found this book a long time ago, but inventing my own solution to OCD was also a fun journey for me.
   • Invisible Masters: Compulsions and the Fear that Drives Them by George Weinberg
   • • Excellent series of stories of real people dealing with compulsions, as told by the author/psychotherapist who’s trying to treat them. Between stories, he also includes short chapters on some of the psychoanalytic theory behind compulsive behavior.
   • Already Free: Buddhism Meets Psychotherapy on the Path of Liberation by Bruce Tift
   • • Excellent book that informed me about ACT and embodying negative feelings. Great read for anyone who has general anxiety, or for anyone who is interested in how western treatment of mental health differs from eastern Buddhism.
   • This blog post: Confidence Engineering: Or, Metacognitive Therapy For Social-Romantic Anxiety
   • • I’ve read this blog post about MCT at least two dozen times. (It’s also pretty funny, in my opinion). Incredibly useful for anyone who has social anxiety and wants to understand how their anxious behavior is (unintentionally) making their anxiety worse.
   • Another great blog post: Social anxiety isn't about being liked
   • • It explains that the reason people feel socially anxious is because they’re deathly afraid of social disapproval. But guess what? You’ll be okay if some people don’t like you—they weren’t meant to be your friend. You can move on.
   • Triggered: A Memoir of Obsessive-Compulsive Disorder by Fletcher Wortmann
   • • This is a college student’s memoir of growing up and dealing with OCD symptoms. It’s brutally honest and made me feel seen. The author also experienced hospitalization for his symptoms (something, fortunately, I’ve never gone through). He’s quite sarcastic which makes it a fun read.
   • Having aphantasia (which I’ve written about) prevents me from experiencing disturbing, intrusive mental imagery that other people with OCD have to reckon with. (Here’s a YouTube video displaying what that’s like).
   • TED Talks
   • • College student with OCD
     • • She talks about surviving a few other common subtypes called Homosexual OCD and Pedophilia OCD—the (unfounded) fears that she may be homosexual or a pedophile.
     • The mom of a child with OCD
     • • Great explanation for how to talk to children about the OCD monster inside them.
   • Life Inside My Mind: 31 Authors Share Their Personal Struggles - edited by Jessica Burkhart
   • • Everyone with OCD follows the same pattern of fear, optional obsession, and compulsion. But how it manifests is always different. These authors’ experiences were different from my own. 
     • Additionally, it was interesting to hear testimonies of people with other mental health disorders besides OCD.
   • Turtles All the Way Down by John Green
   • • John Green is one of the OGs of YouTube with his brother Hank Green, together they started the channels VlogBrothers and CrashCourse. He’s also written various other YA novels that have gotten critical acclaim, such as The Fault in Our Stars. He also has OCD, and wasn’t diagnosed until he was an adult (like me). 
     • The reason I like this novel is that the main character, a teenage girl with OCD, is going on an adventure with her friend. OCD is merely a part of her, but does not define her. It’s a fun story.
   • If you enjoyed this post and want more, consider reading the longer narrative version that explores the inner workings of my mind.

Discuss

(With apologies to Sean Herrington, who deserves a better playwright than yours truly)

A conversation with a friend on the bus to Bodega Bay today made me realize that there are some holes in my thinking about safety and superintelligence. I’ve assumed that superintelligence is by definition robustly better than humans at all the things, but there are some cases when that’s not the case.

Without further ado, for your edification and discomfort, The Strawman Players present:

A Disquieting Conversation on a Bus

Vulpes: I’ve been worrying lately about well-aligned superintelligence.

Corvus: That seems like a strange thing to worry about.

Vulpes: You’d think so. But hear me out. I’m imagining a world where we develop a well-aligned superintelligence (let’s call it MegaBrain) that is omni-benevolent and wants only nice things for us.

Corvus: I notice in myself a distinct lack of anxiety.

Vulpes: But here’s the thing. As part of its mission to serve humanity and give us nice things, MegaBrain develops a cool new technology to make our lives better. The details don’t really matter—for the sake of argument, let’s say it invents a Black Hole Reactor that uses micro black holes to generate infinite clean energy.

Corvus: Still not feeling anxious.

Vulpes: What if MegaBrain is smart enough to develop the Reactor, but too dumb to use it wisely? Perhaps it doesn’t realize that eventually some of the black holes will escape and gradually eat the earth. By the time anyone realizes, it’s too late and the earth—and humanity—are doomed.

Corvus: Ah: I see your mistake, friend Vulpes. You have made the common error of not understanding what “superintelligence” actually means. People often make the mistake of thinking that a superintelligence will be like a mad scientist: brilliant in some ways, but shocking dumb in others. But that isn’t how it works: by definition, “superintelligent” means better than humans in every possible way.

MegaBrain, being superintelligent, will be good not only at designing new technology, but also at understanding how to deploy it safely. If a human could figure out a safe way to test the black hole reactor before deploying it, then MegaBrain could do even better.

So there’s absolutely nothing to worry about.

Vulpes: I’m not sure that’s actually true. What about the jagged frontier?

Corvus: What about it?

Vulpes: AI capabilities are likely to be jagged even as they increase. So perhaps MegaBrain can invent the Reactor because it has superhuman intelligence, but it makes a catastrophic mistake during deployment because it has subhuman wisdom.

Corvus: I suppose that’s possible, but it doesn’t seem very likely. The capability frontier is jagged, but it’s moving fast. Surely there will only a brief period of time when MegaBrain is smart enough to build the Reactor, but unwise enough to deploy it prematurely. There’s only a tiny window of time when anything can go wrong.

Let me revise my earlier statement: there is almost nothing to worry about.

Vulpes: I just thought of another problem.

Corvus: I’m sure you did.

Vulpes: Here’s the thing. Imagine that MegaBrain is installed at the Department Of Maximum Energy (DOME), and DOME is excited to find new energy sources. So they put MegaBrain to work on designing the Reactor, but when it tells them it would be too dangerous to deploy, they ignore it because they’re too eager to deploy this cool new energy source.

Even though MegaBrain is superhuman in every possible way, DOME plus MegaBrain collectively have a jagged frontier. Together, they are smart enough to design the Reactor, but foolish enough to bungle the deployment.

Corvus: Ah. That feels… uncomfortably plausible.

Vulpes: How’s your anxiety doing?

Corvus: Are we there yet?

Discuss

Yassine Meskhout asked for an explanation of the recent emergence of anti-Zionism as a left-wing litmus test, especially salient in the aftermath of the October 7th attack in late 2023. In an unrelated conversation, a friend asked me what recourse ordinary people are likely to adopt as the state breaks various promises to them. My answer to the latter developed a surprising answer to the former.

Our system of government has two interlocking features: it refuses to hear an individual making a reasonable argument, and it systematically disrupts collective threats that fall outside mainstream coalition politics. The result is that reasonable arguments about individual circumstances get drowned out by a competition between acceptable collective identities threatening their rivals.

Under some historical conditions, individual moral appeals could drive real change. Under Quaker norms in Pennsylvania, someone making the simple argument 'you wouldn't like it if someone did it to you' could gradually build support for abolition. People came to see slavery as sufficiently similar to war that they chose to stop inflicting it on others.

On the other hand, consider this story shared by Zvi on his Substack:

I was told a story the week before I wrote this paragraph by a friend who got the cops called on him for letting his baby sleep in their stroller in his yard by someone who actively impersonated a police officer and confessed to doing so. My friend got arrested, the confessed felon went on her way.

As a close friend of the victim, I know additional details. When consulting legal counsel, he was explicitly advised that judges or prosecutors wouldn't respond well to statistical arguments about actual risk levels in child endangerment cases. His lawyer advised him to take a nonsensical online parenting class from a community-service-hours mill as a form of groveling. When he told friends about the incident, many offered advice about what he should have done differently or how to protect himself from authorities in the future. Some suggested trying to get the crazy lady in trouble. No one suggested trying to get the cops in trouble. No one's response was 'That's outrageous - I'm going to tell my friends and we're going to write to City Hall about this.' No one saw it as an opportunity for collective action on the basis of a shared interest as rational individuals - only for individual adaptation to avoid future persecution. [1]

Consider another example: I myself once needed to change my toddler's diaper in a public library. Finding no changing table in the men's room, I had to decide whether I could safely use the women's room. I ended up deciding that I could get away with it if challenged, by declaring my gender "whatever is consistent with being able to care for my child". But it's messed up that that's the first place my mind went, instead of appealing to reason directly, like, "this is a necessity for anyone caring for a child so it's crazy for me not to have access to it on the basis of my sex."

You might think there are other ways to address these problems - changing laws through democratic process, or appealing to universal principles. But any such attempt must come either from an individual speaking as such, or from someone speaking as part of a collective identity. In my experience from childhood onward - and I would love to hear the good news if yours is different - it is rare for individual appeals to reason to work against even apparently slight political forces. [2] Instead, they stonewall, either pretending not to be able to understand, offering wildly inappropriate remedies (like a vendetta against a crazy lady), or if called out clearly and persistently enough, becoming angry at the complainant for presenting such an uncomfortable complaint.

In practice, identities maintain power by credibly threatening collective action, often via state mechanisms. Consider the importance of "protected categories" in civil rights law. Aspects of individual interests not represented by such collective identities get systematically ignored.

Martin Luther King Jr held America accountable to its own founding ideals and biblical principles - reminding the nation it had already explicitly committed itself to human equality and dignity. Yet even these appeals to honor stated principles weren't enough on their own. It wasn't just the logic of King's words that commanded attention - it was the thousands of people marching behind him, the economic pressure of boycotts, and the constant threat of cities erupting if change was denied. This combination of moral appeals backed by collective power didn't just win specific concessions - it established principles and precedents that could be used by others with analogous claims. The Civil Rights Act's ban on discrimination became an intellectual framework that women's groups and others could adapt for their own struggles. The power got attention, which allowed the appeals to reason and precent to create further lasting precedents.

Roughly as the 1960s ended and '70s began, America shifted to a radically lower-trust regime, [3] and appeals to shared principles lost their power. Without the intellectual work of appealing to reason, collective action increasingly produced zero-sum adjustments rather than reusable principles. Dissident identities increasingly turned to organized violence by the 1970s. The FBI's response was methodical - from assassinating Fred Hampton in 1969 as the Black Panthers tried to combine community programs with armed resistance, to developing general techniques under COINTELPRO for preventing such combinations. These FBI counter-intelligence programs included infiltrating groups to sow internal paranoia, disrupting attempts to form coalitions across constituencies, and specifically targeting leaders who could bridge different communities. Any group that successfully combined collective identity with political violence, especially if they showed promise in building broader support through community programs, became a priority for neutralization. Today's dissident identities largely work within the system rather than threatening it - if you can't beat them, join them.

The result is a kind of forced atomization. Modern alienation manifests as either diffuse anxiety/depression or sporadic individual violence (see Adam Lanza Fan Art). Some researchers have suggested intelligence agencies may have influenced these patterns - from documented CIA behavior-modification programs like MKULTRA, to the complex ways intelligence agency actions (like the handling of Ruby Ridge and Waco) shaped domestic political violence. The transition from organized domestic terrorism to serial killers seems to line up about right with the CIA's documented often-lethal terror hijinks. But whatever the cause, our vital national resource of violent weirdos has been successfully redirected from participation in shared identities with a dissident vision for society, to individual acts of psychotic violence.

Anti-Zionism offers the coalition of the left what the system usually prevents: a collective identity that commits political violence. It reads as leftist both because it's anti-Jewish (Jews being, if not white, at least excluded from the POC categories most strongly protected by the left) and because it's explicitly opposed to the legitimacy of nation-states. If you can't beat them, beat them. The fact that Hamas developed its ideology abroad meant this identity could form outside the reach of FBI disruption techniques. The movement's rapid adoption as a left-wing qualifier follows directly from this unique position - where domestic attempts at forming such identities would have been disrupted early in their development, this one arrived fully formed. That this political violence was simply imported and adopted as a left-wing identity marker, rather than arising from any strategic thinking about leftist goals, suggests we're seeing product-market fit for politically-coded acting out rather than a movement likely to achieve anything interesting or constructive.

1. Trying to get the crazy lady in trouble might seem like a counterexample, but the main systematic problem people would have a shared interest in addressing was the behavior of the cops, not the behavior of the woman having a very bad day because her car had broken down. The impulse driving that suggestion was not so much an attempt to solve the external problem, but an attempt to resolve the cognitive dissonance of being asked to acknowledge an injustice performed by a party too powerful to retaliate against, by redirecting the urge to retaliate against a weaker party; pecking order dynamics. ↩︎

2. The civil courts still seem to do this, but decreasingly, after relentless propaganda against the idea of legal liability in the 1980s and 1990s led to the imposition of new legislative limits on tort liability, and a drastic decline in the utilization of civil courts in the US. "Fewer than 2 in 1,000 people … filed tort lawsuits in 2015 … That is down sharply from 1993, when about 10 in 1,000 Americans filed such suits" ↩︎

3. MLK was assassinated in 1968. Fred Hampton was assassinated in 1969. The US exited Bretton Woods in 1971, after which there is no unambiguous economic evidence of net widely shared value creation. See also Have Americans Become Less Violent Since 1980?. ↩︎

Discuss