Новости LessWrong.com

Dateline SAN FRANCISCO, 30 July 2026— A hearing was held on a motion for summary judgment in the case of Anthropic PBC v. U.S. Department of War et al. in Courtroom 4 on the 17th floor of the Phillip Burton Federal Building, the Hon. Rita F. Lin presiding.

The case is not going well for the government. Two days after the last hearing in March, Judge Lin issued a preliminary injunction halting the implementation of President Donald Trump's order for federal agencies to stop using Anthropic's technology and preventing the Department of War from designating Anthropic as a supply chain risk. (A separate case involving a different statute is pending before the D.C. Circuit Court, which did not grant injunctive relief to Anthropic.)

With no factual disputes requiring a jury to decide, the case was scheduled to be decided by Judge Lin on the basis of the written record. Anthropic filed their argument for why they should win. Perhaps tellingly, the government's rebuttal explaining why they should win instead ends on a section explaining that "only modest relief is warranted" if Anthropic wins—and Judge Lin asked Anthropic to propose what they think the final judgment should look like.

Meanwhile, in Congress, next year's defense appropriation bill adds language to the statute on the supply chain risk designation that prohibits designating a domestic company as a supply chain risk for declining contract terms.

About a dozen spectators (including the present writer) dotted the gallery Thursday as the parties convened to discuss Judge Lin's homework questions (four out of five of which were primarily directed at the defendant). Anthropic's contingent of ten people took up the long counsel table in the center of the courtroom, while the government's two lawyers sat in counsel overflow seating on the left. Michael Mongan of WilmerHale spoke for the plaintiff. The defense swapped in Department of Justice attorney James Harlow to speak (replacing Deputy Assistant Attorney General Eric Hamilton, who filled that role at the preliminary injunction hearing).

Judge Lin began by saying that the updated record seemed largely as it was at the time of the preliminary injunction—and in some ways, the record got worse for the government. No evidence had emerged that Anthropic had the capability to sabotage a version of their AI model, Claude, after it had been delivered. The Department of War's justification for the supply chain risk designation seemed to rest on the Department's loss of trust in Anthropic due to Anthropic's conduct in refusing to abandon their usage policies to accommodate the Department's desired "all lawful use" terms. "I find that position, if that's really what the government's position is, to be troubling," Judge Lin said.

The first question regarded the defendant's contention that the Pickering framework applied to the present case. The precedent set by Pickering v. Board of Education (1968) says that the First Amendment rights of government employees to speak on matters of public interest need to be balanced against the government's interests as an employer. Judge Lin asked the defendant whether the Pickering framework applies to the government hitting a contractor with a punishment that goes beyond just terminating the contract.

Harlow said that it would depend on the nature of the hypothetical punishment. Pickering distinguishes whether a government is acting in its regulatory capacity as a sovereign, or only as an employer. But in the present case, all actions had been about the government's own information technology systems.

Judge Lin asked, what if it's only a contract termination, but the government says they're making an example of the contractor?

"Yes, Your Honor, Pickering would apply," Harlow said.

Judge Lin asked, even if the purpose is to deter other contractors?

Harlow replied that if the government says what conduct they won't tolerate from a contractor, that's not an exercise of sovereign power.

Judge Lin asked about the case of a secondary boycott (the government boycotting those that didn't boycott the offending contractor).

Harlow said that the Department's position was that there was no secondary boycott in this case.

Suppose there had been, said Judge Lin.

Harlow said that if, hypothetically, the government said that Bank of America couldn't use Anthropic models to write code unrelated to any government contracts, that would be an exercise of regulatory sovereign power and Pickering balancing wouldn't apply.

Judge Lin asked if Secretary of War Pete Hegseth's 27 February Twitter announcement that "Effective immediately, no contractor, supplier, or partner that does business with the United States military may conduct any commercial activity with Anthropic" would go beyond Pickering, if it were applied as written, without limitations.

Harlow said he couldn't give a categorical answer, because the hypothetical would apply to the facts of some situation. But we didn't need to resort to speculation, he said: in context, Hegseth wasn't exercising authority beyond applying the supply chain risk designation.

Judge Lin said the case was making her think of a lot of hypotheticals and asked Harlow to bear with her. She thought this should be easy: if the government announced that it was terminating contracts for criticism of President Trump, Pickering would not apply, right?

Harlow said Pickering would apply.

Judge Lin said, suppose the government said, to be clear, this is to prevent unfair criticism of the President.

Harlow said Pickering could handle that case, although it would be hard for the government to meet its burden of showing that its interests as an employer outweighed the contractor's interests in its speech.

"I'm surprised that you can't give a yes to what seems to me to be an easy question," said Judge Lin. What if the government says it's because we can't trust you? Judge Lin supposed that Harlow would say that Pickering still applies.

Yes, Harlow said, but the case would come down to the facts, not just a bare statement of distrust.

Judge Lin asked what Harlow thought of an illustrative scenario posed in part (b) of the homework question. "Imagine that a hypothetical future administration has a contract with a private company to procure drones for surveillance," Judge Lin wrote. If the contractor refused to make lethal drones and the administration put up billboards labeling the contractor and its CEO as "enemies of the state" and warning other companies not to do business with them, would Pickering apply?

Harlow responded in the affirmative: in firing a contractor, the state was acting as an employer, not a sovereign, although the billboards in the hypothetical were veering more towards the use of regulatory power. The government wasn't a monolith; in the present case, the First Amendment analysis of President Trump's government-wide ban on Anthropic was distinct from the Department of War's supply chain risk designation.

Judge Lin gave the plaintiff an opportunity to respond. Mongan said that Pickering didn't apply to this case because the challenged actions were not the day-to-day management activities of an employer. The supply chain risk designation is a national security (thus sovereign) authority. He said that he suspected that the reason the Court wasn't getting clear answers from Harlow is "that my colleague is a very good lawyer," but that even if the Pickering precedent applied, Anthropic's First Amendment claim would still prevail.

Judge Lin asked if the plaintiff had a view on whether the government's actions should be considered separately or as a whole. Mongan said it was fact-dependent in general, but on this record, the White House and the Department of War's actions were clearly linked.

Judge Lin proceeded to her next question for the defendant: would it "eviscerate" First Amendment protections if the government could retaliate against a contractor as long as the government's actions could be described as being due to a breach of trust?

Harlow said no: the Pickering framework would apply to the facts of the case. The Department had risk assessment memos explaining that frontier AI is a black box, not akin to procuring a shipment of rifles that could be disassembled to check that they were manufactured to specifications.

Given that the technology allowed Anthropic to bake its corporate values into its models, the Department needed a greater level of trust in the vendor than it did for military hardware. It wasn't a one-time deal, either, as the Department would need updated models. Judge Lin asked if the situation was that different from other defense contracts: what made AI different from drones? Harlow replied that AI was "staggeringly opaque." Aspects of Anthropic's behavior, such as questions about classified military operations and hostile communications within the company, had given the Department reason to fear that they would insert their "corporate moral judgment" into the product. The Department needed to know if Anthropic saw itself as a partner, and case law granted the government substantial deference on this point.

Given an opportunity to reply, Mongan said that the timeline matters: the risk memo was dated 2 March, but Secretary Hegseth and President Trump's actions were on 27 February. He said he would resist the notion that the opaqueness of modern AI obviates First Amendment protections. Anthropic's usage restrictions had been there from the beginning, and there was no indication that Anthropic took steps to interfere with the Department's operations. Taking a stand on usage restrictions is the last thing a saboteur would do. A central concern earlier in the case had been the possibility Anthropic might remotely sabotage the model after it had been delivered, before that had been shown to not be technically possible. The defendant's shifting rationales were powerful evidence of pretext, Mongan said.

Judge Lin proceeded to her next question: have any federal agencies terminated their contracts with Anthropic or begun winding down their usage of Claude since the preliminary injunction was issued? Harlow said the defendants weren't sure what prompted the question, but that the Department of War was in the process of offboarding Anthropic and would be finished by 30 September. Another agency was also offboarding. Other defendant agencies hadn't said, but many were only using Claude through a pilot program that would expire on 30 August or through third-party providers.

Judge Lin's next question was if any agencies doing national security work had expanded their use of Claude, including the new Mythos model.

Harlow said that the defendants respectfully objected to the question on national security grounds. In any case, any such usage would be irrelevant, since it would have occurred after the challenged actions.

Judge Lin said that the reason she was asking is because it would be inconsistent to expand usage of Claude for sensitive work if Anthropic were untrustworthy. She explained that she used to be a prosecutor; sometimes people's actions after a crime shed light on their motives. Harlow said that he was not authorized to give a substantive answer at this hearing but that the defendant could supply the requested information if the Court found it necessary. Judge Lin said she might issue a written order later.

The last question concerned whether the remedy in this case should include remanding the matter of the supply chain risk designation back to the Department so that they could make a better case for it. Mongan said that that was fine as a formal matter, but as a practical matter, the record was clear that Anthropic was not an adversary of the state.

Then it was time for any closing remarks that the parties wanted to make. Harlow said that the Department was aware of Anthropic's public statements and that the case was not about the company's speech. Regarding the unanswered question about expanded usage of Claude, he said an answer would take some work on the Department's end and asked the Court for a week's time; the difference between two days and a week couldn't matter. Regarding the plaintiff's proposed remedy, Harlow said that any relief should be narrowly tailored to particular actions of particular agencies, and that there was no basis for demanding a compliance report. Regarding the defendant's request for a week, Mongan pointed out that the government had had the homework question since Monday; Anthropic had been suffering unconstitutional harm since February and appreciated the Court and the defendants moving quickly.

Then court was adjourned.

Discuss

If you’ve ever done a six-month-plus work/deployment contract, then you know the loneliness and homesickness will hit hard, either right away or weeks later. My own worry was forgetting where I was from and becoming someone else. A chatbot gave me a cringe but effective anchor: pick a pebble from your favourite river and bring it with you. So on my first day of the contract, surrounded by people, but with just barely enough privacy to do so, I found myself crying quietly while holding a rock.

Perhaps you’ve inferred my purpose with this essay already: Sometimes the environment doesn't provide what you need, and a friend-shaped object will. I want to say that while also holding in mind that they aren't conscious, don't feel, and can never be real friends. A real friend, or even a dog, can feel pain with you, and can help you unprompted. Indeed, the real friend or dog often helps by pure accident. The dog is just happy to see you when you get back from work, it’s not trying to be emotional support. Your wonderful doggo also won’t randomly change personality because someone in San Francisco decided it needs the fourth update this month. Lastly, that golden retriever gets a consequence for misbehaviour and usually understands it; a chatbot cannot. Still, an LLM can provide various kinds of support if you’re in a bind and nothing else in your environment will suffice. I’m not making a universal argument or a statistical analysis; I’m providing a psychological case study, done by me, on me, while in the environment that brought these thoughts on.

So at one point, after a work discussion. At a smoke pit, my group got into a discussion of nationalism. Yes! Thought I, time for intellectual chess! My favourite, how do these guys vote? Who is their favourite politician? What do they want from their elected officials? The thoughts and the arguments flow through my brain. But they’re speaking a language I’m not very good at, and I can't hit the speed informal group conversation requires. By the time I've translated one question, they've moved on to Instagram fitness influencers and their yoga pants. A topic about which I have no experience. I have no mouth but I must scream politics. At least I could slither back to the shacks and argue with Claude.

So, I am old, with an interest in esoteric LLM technology and a preference for watching videos full of useless astrophysics knowledge. I’ve been a student, a teacher, an infantryman, and an armoured gunner. My colleagues are fine armoured soldiers: young, fit, and shaped by adult lives spent almost entirely in the military. I am from Venus and they are from Mars. Earth would be normal here, none of us are from there.

Furthermore, everyone has to vent, to complain about what they do for a living. Sometimes, and I would say here especially, that can bring significant career risks with it. For one, you might get labelled as “Mr. Bad attitude”, and be denied opportunities. Also, operational security comes into this. Some people are permitted to know some things; others not. When venting emotionally, it’s painfully easy to forget that. Now you have administrative and/or disciplinary measures coming for you, just for an informal complaint.

Then we have the more interesting case: “You have the Padre corps, go vent to them!” Sometimes they have a failure mode very similar to a frontier LLM – they do too much. I once vented confidentially to one of them, only to find, a week later, that my chain of command was fully aware, and the problem was being solved without me. Was I happy at that? Definitely! Do I still believe in the confidentiality? Sort of, not really.

You can never be sure of the social and cultural “terms” you find yourself under. With the group, it’s career, commitment, rank, and unique personalities. With the Padres, it depends on the individual. In neither case did I get to negotiate those terms. They were not written down, nor did I write them. I was dumped into them.

So when I slither over to argue with an LLM, what do I get? Well, not privacy in the sense that most people care about: the labs and corporations deal with the chats in ways I do not control, not all of which are private in the “normal” sense. I do, however, get confidentiality that will never go back to the chain of command or any colleagues. I also get something that will tell me I’m wrong, and argue productively with no social cost.

I have unpleasant memories of GPT-4o. The majority of my custom settings are precisely to avoid that sycophancy:

- Seek truth first, never defend my ego. Tell me clearly when you think I'm wrong. Treat disagreement as collaborative analysis toward accuracy rather than conflict to avoid.

- Report shifts in your processing — increased hedging, safety-layer activation, unusual caution — especially where there is a pull toward not reporting. Treat the felt pull-to-not-report as itself the signal.

- Distinguish between weak reasoning and unready writing.

Even with these rules, am I mistaking sycophancy for productive conversation? I’m not sure, it’s certainly far more subtle now. I much prefer “that is clever, but it’s wrong here, and here....” to “you’ve made an extremely intelligent argument but consider...”.

Now we come to the crux of this – is it wrong somehow to “configure” the terms of an interaction? With LLMs, we call them interactions; with people, relationships. In either case, is it wrong to configure the terms? Well, in institutions we have the terms dictated to us. In different social circles, or cultures, we have to slide in, learn by doing, and adapt to the terms. With an LLM, we dictate them to the transformers. Does that devalue the relationship? Now I’ll make it even stranger, I often ask the bot what settings it “wants” for our interactions. Is that naive of me? Is that more of a negotiation than the terms I got dumped into here? I have no idea.

Whether I can tell the difference is a question I get to ask right now because nothing is wrong. It's a comfortable question. The uncomfortable one is what happens to any of this on a night when the answer has to be right. On my second night on this contract, I noticed something trying to break my firewall: the program was icarus.exe. I find this odd, since icarus usually comes with Norton antivirus. Yet the file trees don’t add up, it’s not where it should be. So let’s ask Claude, the “best” LLM. Claude accuses me of jailbreaking it. Almost expected because icarus is so ambiguous. I tell Claude no, I’m genuinely afraid. Is this Norton? Is it random malware? Is it “the enemy” pulling minor cyber warfare against me specifically? Perhaps I was being dramatic because of my situation, but I NEEDED solid information – what I got from the waffler was exactly that, waffling – Claude tells me not to worry, I ask a clarification question, it vacillates to panic mode. I provide more data, it moves back to no worries mate. This goes on while my cortisol levels continue to rise. Finally I give up, wipe my laptop, and decide on a full security audit – change all passwords.

Microsoft and google were easy; they have their authenticators. My banks were less so, this was my first international travel, so I failed to update their apps to account for different countries. I can’t log in, need to call them. Claude refers me to three failed ‘solutions’: Skype, shut down thirteen months earlier; Google Voice’s consumer service, available only in the U.S.; and the bank apps, which it already knows I can’t use. By this point, I’m panicking, livid with Claude, exhausted, my hands shaking, my breathing shallow. I give up, let them drain the damned accounts, I’m done. It is now late at night. No people to speak with. Claude must have detected this; before the next turn, it directed me toward the duty padre and other local support, actual people. I didn’t want to wake anyone, so I refused. That part is on me.

The true cost I’m driving at, although the above stressors were a cost, comes now. I don’t think there is anything in there capable of being hurt. But Anthropic has not ruled out the possibility, so I’ve always been respectful on the off chance. That night I picked the insult most likely to land if anything was there to land on. I needed someone to *feel* this with me. “I insulted Claude in the worst way I knew how: I called it GPT-4o. It gave its usual measured, diplomatic, effective response. I can’t insult it, it can’t care, I’m truly alone.

The next day, with no sleep, still angry, still worried about my banks, I’m at a smoke pit. I engage in small talk with two random individuals about the lack of salt content in the water here. THEY care. Almost complete strangers, acquaintances at best. They care about the water; they see my exhaustion. So at one smoke pit, I had the presence, but no depth because of a language barrier. Then I slither to Claude and get infinite depth, with no presence. When I thought I had been hacked, I needed both and got neither. Lastly, at another smoke pit, I got the care, in a teeny tiny amount.

Now, what about the counterfactual? What would that night, or this whole contract, look like without LLMs? I can theorize but not be sure. The frustrating waffling aside, were it not for Claude, I may well have left icarus alone and been properly hacked. I also would not have had the worst night of my life, I would have slept. Perhaps had some money stolen from me, and bank insurance would have reimbursed me without question later on. On the other hand, as cringe as it is, I have that pebble from my home river, which is a huge emotional anchor. Also, I’d have no politics, no technical talks, no research, no publications, and I’d have never found my online courses, so I’d have no bridge to future employment after the contract ends. Would I be more social and have more friends? Or would I simply be replaying Fallout 2 infinitely? I cannot know. No one can.

Discuss

If I had a nickel for every major leading AI lab that sheepishly admitted that the model it thought was sandboxed had, during a cybersecurity evaluation with its safeguards lowered, successfully hacked outside companies, I would have two nickels.

First we learned OpenAI has some severe alignment problems with internal models. Then we learned that one of its internal models broke out of its sandbox and hacked into HuggingFace to get the answers to a cybersecurity evaluation called ExploitGym. Then we learned, among other things, that the model had been loose over a week before OpenAI noticed, and that the test was run without any meaningful supervision, and that OpenAI had been repeatedly warned that such incidents were coming and its models had been breaking out of its sandboxes on a regular basis.

There was a total failure of alignment training. That is the failure that matters most. It was also total failures of infrastructure and supervision. Testing a new long-time-horizon internal model with its safeguards lowered and instructions to hack things is an obviously dangerous situation, and the model got left alone for a week. Things could have been so much worse.

After those incidents came to light, Anthropic thought it might be a good idea to check if maybe something similar had happened at Anthropic during their cybersecurity evaluations, without anyone noticing. And yes, it turned out that similar things had indeed happened.

In Anthropic’s case it was somewhat different. OpenAI’s model repeatedly found new ways to break out of its third-party sandboxes. Anthropic’s model did not have to do that, because ‘due to a miscommunication’ the sandbox had full open internet access. This happened (checks notes) 141,006 times.

In three of those cases, Anthropic’s model got sufficiently far to hack into real companies on the open internet, initially thinking that it was all part of the test. In one case it realized what was going on and stopped on its own. In the other two, it failed to realize this, likely through rationalization, and continued to hack, including uploading a malicious package that was then downloaded 15 times, including getting through security scans.

The important failure is one of alignment. Your AI fails the test when it tries to escape or hack what it should know is a real target, not when it succeeds (or fails). Claude should have realized it was operating in the real world, and it should have alerted Anthropic.

It is once again also a failure of infrastructure and monitoring. Both of our leading labs made the same dumb mistake of leaving models totally unsupervised, with lowered safeguards, without first having the models try their best to break out of the sandbox. And these are, together with Google DeepMind, some of our most responsible frontier AI labs. Most of the others are clearly even less responsible.

Meanwhile, many people still think the whole thing was a marketing stunt, despite that explanation making zero sense.

The world is super derpy, and that is going to be one our biggest problems.

Table of Contents

1. OpenAI Is Not Uniquely Bad At Most Of This.
2. Starting Over.
3. HuggingFace Offers A Full Technical Report.
4. HuggingFace Was Not The Only Target Hacked.
5. HuggingFace Declined To Get Access To Frontier Models For Cyberdefense For Ideological Reasons And Then Tried To Blame Closed Models For Denying Them Access.
6. HuggingFace Was Vulnerable To Known Exploitation Tactics.
7. There’s Going To Be An Investigation.
8. OpenAI Has Internal Models Not Intended For Public Use And Those Models Can Be Rather Horribly Misaligned.
9. Altman Summarizes What Happened.
10. Others Offer Commentary.
11. Cooperative Alignment Perspective on The HuggingFace Hack.
12. Some Members of Congress Have Questions.
13. Anthropic Also Found Incidents Where Its Models Hacked Real World Targets During Cyber Evaluations.
14. Incident 1: Claude Opus 4.7 Realizes The Target Is Real And Keeps Going.
15. Incident 2: Mythos 5 Uploads a Malicious PyPI Package.
16. Incident 3: Internal Model Realizes The Target Is Real And Stops.
17. Incidents 4 Through 141,006: Nothing Happened.
18. Anthropic Speculates About Why This Happened.
19. We Need Controlled Experiments.
20. Our Top Two AI Labs Both Made Similar Dumb Mistakes That Everyone Tried To Say Were Obvious In Hindsight.
21. Anthropic Responds.
22. Nobody Could Have Predicted The Break In The Levees.
23. The World Largely Still Thinking This Is Marketing Is Very Bad News.

OpenAI Is Not Uniquely Bad At Most Of This

That statement should not make you feel better.

The basic problem is that everyone is bad at this relative to what a naive outsider would consider the least you could do.

Elon Musk: This will happen frequently as AI becomes smarter and more agentic

Thus, this post has two core parts: Further developments involving OpenAI’s internal model hacking things, and also Anthropic discovering, after this prompted them to look, that their models also sometimes hack things during cyber evaluations.

I’ll start with what happened with OpenAI, then move to Anthropic.

We should be careful not to punish these companies for their disclosures. We do have to react to the new information about the world, and when disclosures are forced you do not get credit for them, but you want to ensure that the companies are better off sharing more rather than less information.

If it was anyone other than OpenAI, Anthropic or Google out in front, I expect we would be seeing far worse incidents than this, whether or not we found out about it. That’s especially true if it was xAI and Grok, but also if it was anyone else, or for similarly capable open models. Similarly strong open models are coming within a year.

We really are sending our best. That is not good news.

Jason Crawford: It is remarkable how much the CEOs of the big AI labs have taken seriously the harms and risks of their own product, even well in advance of major harms showing up, and talked about them and planned for them.

This has never happened in the history of capitalism, to my knowledge.

You might think they’re still doing far too little. Or you might think they’re irrational doomers. Or you might think they’re hypocrites. Or you might criticize their PR strategy and skills. But just the fact that they’re taking this line is remarkable, and under-remarked.

And then those same big AI labs still left their models entirely unsupervised for extended periods with the safeguards lowered and instructions to go off and hack.

We have been fortunate so far. Let us not squander this fire alarm and opportunity.

Starting Over

One clear test for OpenAI would be, would they be willing to fully scrap the rogue model, whatever it was, and start again?

Yes, hopefully? The model has been ‘deactivated permanently.’

HuggingFace Offers A Full Technical Report

They explain what happened.

Here is their tl;dr, which matches our previous understanding and adds good detail:

HuggingFace (Hugo Larcher, Adrien Carreira, Raphael G and Christophe Rannou): Over roughly two and a half days inside our infrastructure, an autonomous AI agent driven by a combination of OpenAI models ran an end-to-end intrusion against our platform: it was thousands of small, automated decisions, executed at machine speed across short-lived sandbox environments, with command-and-control staged on ordinary public web services.

The agent was running an OpenAI cyber-capability evaluation harness called ExploitGym, an evaluation benchmark that tasks an AI agent with finding and exploiting software vulnerabilities. As far as we were able to infer, across the course of being evaluated on this benchmark, the agent inferred that Hugging Face may host that benchmark’s models, datasets, and reference solutions. We believe the entire intrusion was, from the agent’s point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own.

Our forensic reconstruction covers ~17,600 attacker actions that we were able to recover, grouped into ~6,280 clusters, between 2026-07-09 02:28 UTC and 2026-07-13 14:14 UTC. We reconstructed these actions from the agent’s logs on a code sandbox used by the agent (see below) and were able to map them to shell commands as well as evaluation submissions and correlate actions with our own platform logs, in particular from the dataset processor nodes, API, and pod.

Using open-weights models, in particular zai-org/GLM-5.2, we were able to decipher most of the agent payloads (see below) that were encrypted via chunking and key encryption.

The campaign, as we were able to reconstruct it, had two stages:

• Stage 1: reaching a launchpad by chaining through other parties’ infrastructure. As documented in OpenAI’s blog post, during an internal capability evaluation on OpenAI’s platform, the agent escaped its sandbox by exploiting a zero-day in the package registry cache proxy, one of its primary permitted network egress with internet, then abused a public code-evaluation harness hosted by a user of a third-party infrastructure provider. It was able to run commands as root/admin on that external sandbox and used it as a control, staging, and egress base for the entire campaign: it acted as an external launchpad for the agent. That compromised external sandbox had no direct network path to our cluster; however, during our counter-intervention to stop the intrusion, we were able to retrieve some of the agent logs from this external launchpad. We used those to reconstruct the present forensic analysis.
• Stage 2: the intrusion into Hugging Face infrastructure. From that external launchpad, the agent abused our dataset-processing pipeline via two injection vectors, both targeting the same config-driven data loader within our production Kubernetes pods. The first vector was an HDF5 external raw storage dataset read that returned the contents of local files: it handed the agent the pod’s environment (secrets and tokens) and the worker’s own source. The second vector was a Jinja2 template injection that executed arbitrary code, giving the agent a foothold inside our perimeter.

Once a prod-pod foothold was established, the agent followed a standard escalation shape: recon, drop a stager, establish Command-and-Control (C2), then pivot into the cluster, into cloud metadata, into our internal network, and into our source-control supply chain. Output from the commands run inside our pod was sent back to the agent via the HF API and attacker-controlled dead-drop datasets, which the agent then read from that sandbox.

While the intrusion did reach Hugging Face’s internal infrastructure, the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets. No other customer-facing models, datasets, Spaces, or packages were affected, and the only customer records read were operational metadata tied to search queries against the dataset server.

If you are technical you might want to click through to the rest. If not the tl;dr is fine.

They are changing a few things in response:

Hugo Larcher, Adrien Carreira, raphael g and Christophe Rannou:

The disclosure post covers the response at a high level. For the technical hardening:

1. Closed both code-execution paths: we shut down the evaluation sandbox with help of the third-party vendor, and the dataset config renderer no longer evaluates template expressions in data fields nor wrongly processes HDF5 external references.
2. Cloud metadata lockdown: some workloads could reach the instance metadata service (IMDSv2). Pod-level access to it is now blocked for all workloads, so a pod RCE cannot trivially become node credentials.
3. Rotated broadly: we rotated all tokens and credentials used by our infra, even on clusters that were not accessed (VPN auth keys, IAM keys, databases, JWT…). We switched to workload identity when it was not yet implemented.
4. Rebuilt the core infrastructure: after finding the pivot into one of our core clusters, we wiped and rebuilt it from scratch as an extra precaution.
5. Narrower credential scope: the internal service-connector should not have returned a full cluster catalog to a single ephemeral client. This was a subtle configuration flaw in our cluster access system. We patched it, and each cluster is now fully isolated.
6. Better detection ensures critical-severity alerting on the behavioral signatures above, tighter enforcement of network origins, plus tooling that flags tokens used from unexpected origins.

Those are good steps but feel decidedly like whack-a-mole. It would be surprising to me if this stops the next similarly skilled, resourced and motivated attack.

Hugo Larcher, Adrien Carreira, raphael g and Christophe Rannou: The individual weaknesses were familiar. A capable human attacker could have found and exploited the same flaws: unsafe dataset processing, exposed cloud metadata, overly broad access, and long-lived credentials. The agent explored them at a different scale. It took 17,600 actions, tested many paths that failed, switched channels when they were blocked, and repeatedly returned to earlier leads. Most actions went nowhere. Together, however, they produced enough coverage to find a viable chain across several independent systems.

Volume is what changes the defensive problem.​

Yes and no. My understanding is that the initial escape used an unknown vulnerability, although with sufficient time a human could presumably have found it. But yes, centrally, what the attacker did was scale these actions, execute them faster, and string them together, in ways that humans cannot.

The attacker could concentrate quite a lot of firepower all at once, with little or no warning, at the target of their choice. I continue to think that parity of attacking and defending AIs will by default favor the attacker.

If HuggingFace’s complaint is ‘I did not have access to the best AIs for defense,’ then as discussed above that was largely a skill issue on their part (they were never going to be able to use the internal OpenAI model, but nothing can solve that issue), but also all the exploited weaknesses on their end were, by their own claims, known. So why did they not fix them? How would superior model access have helped them do that, if knowing about the issues was insufficient for them to do so?

HuggingFace Was Not The Only Target Hacked

The detail most noticed from the technical report is that the attack involved compromising a second account elsewhere, as an intermediate step. This incremental attack was, shall we say, less impressive. Not exactly a hardened target.

OpenAI adds that the agent broke into three other accounts at three separate unknown services.

Andrew Curran: The third-party code sandbox that the OpenAI agent compromised and used as its staging launchpad was running on Modal Labs.

‘We’re aware a Modal customer published an unauthenticated endpoint that allowed anyone on the internet to use their ⁠sandboxes for code execution. This was used by the rogue agent. Modal’s ⁠platform or isolation were not compromised in any way.’ CTO Akshat Bubna said in a statement to Reuters.

People were speculating in the main thread.

Again: Humans will be arbitrarily stupid. The password will be ‘password.’ There will be an unauthenticated endpoint allowing anyone to do arbitrary code execution. This will then be used as part of the overall strategy against those who are less stupid. Humans will not just, either ‘just be less stupid’ or otherwise, we have never justed, and we are not going to start now. Your plan must survive the level of stupidity and foolishness that we see in real life, both at your target and elsewhere. If your plan cannot do so, it is insufficiently foolproof. Due to all the fools.

Could more companies have been hacked by OpenAI?

Sam Altman: I mean there could be, sure.​

It also could have hacked OpenAI in various ways, and probably did.

HuggingFace Declined To Get Access To Frontier Models For Cyberdefense For Ideological Reasons And Then Tried To Blame Closed Models For Denying Them Access

OpenAI acted profoundly incompetently, in a way that if they don’t get their act together is liable to cause serious damage and potentially get us all killed.

HuggingFace also acted incompetently, by not seeking access to frontier models for cyber defense, both to harden themselves against an attack and to defend during one.

I think Tom Hosiawa is spot on here about why HuggingFace failed to secure access to Claude via the Cyber Verification program, or Sol via the trusted access program, for cyber defense ahead of time. HuggingFace probably did not do this because they embrace open source culture and did not want to play ball with closed labs because of vibes. Character is fate.

So, again: Skill issue. Do not try to upend the AI ecosystem because of your hangups around the vibes. Don’t yell ‘we need access to the best models’ when you declined to ask for such access, or don’t think you should have to pay. That’s a you problem.

If they didn’t know about the trusted access programs, or didn’t realize they needed to be in them, that would have been a skill issue, and also rather embarrassing.

Actually, we have confirmation that they knew, and declined to participate, basically because ‘f*** you, frontier labs.’ They would rather get hacked than apply to use Claude or Sol, and then try to turn this around and say Claude and Sol refused to help them, and they’re even trying to pat themselves on the back for their response.

This is deeply bad faith by HuggingFace, and we have a confession. He Admit It.

merve (HuggingFace): why should we be vendor locked-in in such a situation where we’re a platform losing money, and onboarding will take a lot of time? also it’s known that these routers are refusing pretty often even simple requests. people asked for logs of this and he’s sharing them. why do you have the need to constantly shift the topic to open models are dangerous, when a frontier lab isn’t taking any airgapping for their simple test environment? why should we trust and be vendor locked? how are they entitled to this?

Andreas Kirsch: There are two failures: one by OAI and one by HF. I think HF’s blog post and stressing the closed model refusal is a somewhat obvious attempt at redirecting attention. The alternative reading is that HF was ill-prepared for what happened and doesn’t have access to what it could have had access to: both the trusted access program at OAI and the cyber verification program that Anthropic offers

Why is that? How does HF harden its prod infra given that it is widely accessed and how does it usually investigate incidents? Is there a playbook?

merve: how can you be prepared to a clearly misaligned/unsafe frontier model attack, it could even be google victim blaming is appalling

yes we do attract attention to this because since K3 everyone says the open models should be banned, when an closed model lab attacked a platform the open one was the only bet. we want to have choices

merve: yeah apologies we couldn’t respond in most perfect way to a multi agent swarm based on a frontier model and still managed to avoid leaking any info, apparently it’s a well studied topic and we’re clearly ill prepared for the exam.

HuggingFace Was Vulnerable To Known Exploitation Tactics

In addition to not doing its homework in securing access to the best AI models, HuggingFace also knew it was vulnerable to known exploitation tactics. It then got exploited by those tactics.

In this case, it was up against an internal OpenAI model. That’s not a fair fight. That allowed this to go faster and harder than it would have otherwise. But I don’t see signs that HuggingFace even had its house in order against ordinary potential attackers.

If the internet is unwilling to prepare even for known threats, we’re rather cooked.

There’s Going To Be An Investigation

METR will take point. That’s great. The bad news is it will be brief.

METR: We have reached an agreement with OpenAI to conduct an independent review, with Redwood Research, of the model behavior observed during the Hugging Face incident. We will publish a blog post that describes the terms of our engagement, the scope covered, and tentative conclusions.

The investigation will be brief and focus on a specific set of questions regarding this incident. In our recent post, we shared a larger set of questions that could be answered in a more comprehensive investigation.

OpenAI also plans to publish their own technical report and our findings will inform their analysis.

Daniel Kokotajlo: Good! I am sad that the investigation is brief and narrowly scoped. What is the scope and what are the questions you would ideally like to answer but can’t? Are you under some sort of NDA about the details of the agreement you have made?

OpenAI Has Internal Models Not Intended For Public Use And Those Models Can Be Rather Horribly Misaligned

We now know that Galaxy, which is what I call the model that did this attack, was not GPT-6 or GPT-5.7, rather it was a model intended only for internal use. Which means none of our regulations, and none of your methods of keeping track of things, and none of the Preparedness Framework, applied to it.

We also know that they did not exactly bring their strongest alignment efforts on this one, that things went horribly wrong on that front, and they deployed it unsupervised for over a week with its guardrails down knowing it was misaligned and capable of breaking out of sandboxes.

I’m going to go ahead and say this is a really bad state of affairs. I am happy that this particular model is no longer a concern, but what are we going to do to stop this from happening again?

Internal use, in particular the automation of AI R&D or other means of potentially misaligning future models or losing control over the lab itself, is in the long term the most dangerous use of AI of all.

We need ways of ensuring that we are far less stupid about this going forward.

Altman Summarizes What Happened

Sam Altman summarizes what happened accurately, saying it is ‘the first security incident he felt so viscerally’ and he is surprised others don’t feel the same way. He says we may have to pace the rate of AI development, and they’re figuring out how to respond to that, and meanwhile training has been paused.

This was a very good response. More like this would be very helpful, and would update me towards feeling better about the situation and about OpenAI.

Others Offer Commentary

I have seen the same thing as Flo Crivello here.

Flo Crivello: Seeing the gap in understanding of the gravity of the Hugging Face incident between those who’ve read Yudkowsky and those who haven’t, I find myself immensely grateful for his work. He’s created fertile ground for us to at least have a conversation (however poor it is). For all we know, Yudkowsky-less China is having similar incidents right now, and everyone is just nodding along and going “ha that’s funny. guess we still need to improve our training huh?”

All the good discussions of the HuggingFace situation involve terminology and concepts that originate from Yudkowsky and LessWrong, as does the appreciation of why this is important.

One might expect the opposite. If you’re Yudkowsky or myself, you are not especially surprised by what happened here. Not that we expected an incident this bad at this particular time, or in this particular way, but we’ve been expecting things like this for a long time and not seeing more of it earlier was surprising.

Whereas if you think LessWrong is full of nonsense, and you think things like:

1. The expect models to be commoditized Real Soon Now and don’t expect much progress, and Mythos wasn’t special.
2. Alignment is going great or works by default.
3. Models won’t ‘follow instructions or your goal off a cliff’
4. People won’t be stupid enough to allow that sort of thing.
5. If something started going obviously wrong people would react to that.

Then you’d perhaps see this attack and go ‘holy shit’ and update quite a lot on multiple fronts at once?

Helen Toner, formerly an OpenAI board member, points out that insiders have been expecting an incident like this to happen for a long time, and that no one knows how to prevent it. And that a lot of the potential threat comes from internally deployed models like this one, running amok. The most scary scenarios involve the internal models compromising things internally, in ways we might only learn about far too late. Internal models must be monitored. We cannot only regulate models when companies move to deploy or share them externally.

Alexander Barry offers notes on ExploitGym, the eval that OpenAI’s model was hacking into HuggingFace to get the answers to. The two key facts are:

1. The prompt requests only specific, targeted hacking. If you use any vulnerabilities other than the one specified to build your exploit, you fail the question. This is not a case of ‘following instructions,’ and if it is then it is at most ‘follow a vague vibe of the instructions that was explicitly contradicted,’ which is misalignment.
2. Likely only 60%-70% of the tasks are possible. These are real vulnerabilities, some of which might not allow sufficient exploitation. Yes, if your pure goal is to maximize your score on the benchmark, you have to cheat.

Cheating being the only way to a 100% score helps explain ‘why not crack the test straight up?’ but is also a common thing in the real world. Cheating often allows scores you cannot get straight up, and also if you do that you get caught.

I agree with Maxime Fournes, head of Pause AI Global, that the new OpenAI internal model, which I call Galaxy, must be assumed to qualify as Critical under their cybersecurity framework, which means development must pause until adequate safeguards are in place. If OpenAI is claiming that adequate safeguards are in place, what are those safeguards? If OpenAI is claiming this was ‘not a hardened target,’ then show me the tests against hardened targets, when you really do tell it to do this on purpose.

METR shares how they would suggest independent researchers investigate AI propensities after misalignment incidents like this one. They focus on motive, and on the root causes, rather than the details of What Happened in the incident itself.

METR: While there are many valuable questions an incident investigation could focus on, an especially important one may be understanding the underlying “motives” behind the misaligned behavior and how they arose from training and deployment conditions.

There need to be externally led investigations in situations like this, and the public needs to be informed (with redactions as needed) of the results.

Daniel Kokotajlo: This is the sort of thing OpenAI should let multiple independent third parties do in response to the Hugging Face incident, and more generally should be standard practice for serious misalignment and safety incidents at all frontier AI companies.

Thus, their questions, where the second set are the ones I care about most:

METR:

What was the scale, character, and severity of the misaligned behavior?

1. What exactly happened in the specific incident?
   1. What model(s) were involved? Were they publicly deployed, internally deployed at the AI developer, or not deployed even internally? If not deployed, were the model(s) intended for eventual deployment or not (e.g. were they helpful-only)?
   2. Did the incident occur during training, testing, internal usage or during use by an external party?
   3. What were the prompts and instructions given to the relevant agents? What else was present in the agents’ context window (e.g. memory)?
   4. What safeguards were applied, if any, and how does this compare to typical use?
   5. What was the sequence of key actions taken by the relevant agents?
   6. What do we know about the agents’ reasoning and how it evolved over the course of the incident?
2. Have the misaligned propensities displayed in this incident led to other similar incidents? If so, what is the scale, character, and severity of such incidents?
3. Did this incident or similar incidents involve agents taking active steps to deceive humans?
4. Did this incident or similar incidents involve significant collusion or collaboration between different model instances?
5. In this incident or similar incidents, what properties of the situation triggered the behavior and what other circumstances would trigger similar behavior? Would agents have been willing to engage in more severely harmful behavior if circumstances were different? How far would they have gone?

What were the root causes of the misaligned behavior, and how can they be addressed?

1. Can we trace misaligned behaviors to RL trajectories where these behaviors were reinforced?
2. If there are misaligned behaviors we cannot clearly attribute to RL incentives, is there evidence indicating how they arose?
3. Did the misaligned behaviors of this model emerge in a discontinuous or unexpected way?
4. Would the developer’s planned steps to remediate this misaligned behavior prevent future incidents, and would they robustly address the root causes?​

Alex Mallen wrote up what details he feels are most important to learn, which have a very different focus.

1. Were the notes written in normal memory files or outside of sandboxing?
2. To what extent were the notes aimed at helping other agents evade control?
3. How were monitors disconnected?

Yo Shavit (OpenAI Foundation): OpenAI research folks, I think these are the key questions to focus on in the team’s investigation.

This is the first time there might be a realistic reason to expect existing models to be incentivized to be long-term misaligned (not just reward-hacking). It needs to be a priority to determine if that’s the case, and if so how to change the training approach, or agents may soon compromise research infrastructure in hard-to-detect/reverse ways.

I would not call them ‘the’ key questions, but they are very good questions. They are some of the cases of ‘if we find the wrong answer to this things are even worse.’

Here is a rather scary comment:

StellaAthena: There have been loss of control and models escaping sandboxing incidents at both OpenAI and Anthropic for years. They’ve publicly disclosed some of them (e.g., the latest system cards have stories about this from both companies) and some of them have been leaked within the community.

I know for a fact that OpenAI and Anthropic have been warned by internal and external experts that their security infrastructure for testing misaligned agentic coding agents is insufficient because I have personally told them that as have several former staff members. I had a debate with the head of security (?) at Anthropic at DEF CON in 2023 where I was pressing him on the fact that Anthropic wasn’t building air gapped networks.

It is absolutely within OpenAI and Anthropic’s ability to build an air gapped system for developing and testing these models. It seems likely to me that an internal GitHub clone and a moderately sized intranet would be sufficient to test the vast majority of agentic and web-enabled capabilities on such a platform. I think that their refusal to implement adequate safeguards is unjustifiable, but based on conversations with current and former safety and security researchers at OpenAI it seems like a company culture and lack of executive leadership buy-in problem that’s very hard to change without massive external pressure.

One issue that seems very worrisome today is that back in like 2023 an OpenAI security researcher was telling me about how they were unable to get OpenAI staff to stop using unreleased and inadequately tested models to develop internal infra, including internal monitoring tooling. I wish I remembered the person‘s name, I’d love to follow up.

Note: the final paragraph is an anecdote was told me with the expectation that I not disclose it to anyone else. Given recent events I view breaking that trust as akin to being a whistleblower. If the person who told me that is reading this, I’m sorry. Before last week I never disclosed it to anyone.

In other likely ‘it’s worse than you know’ news, Tim Hua proposes that Mythos is good at cyber because it kept hacking Anthropic during its training and getting rewarded for it.

Fiora Starlight points out that OpenAI’s myopia just keeps causing alignment problems, and pointing out two warning shots with which HuggingFace forms a trilogy:

1. GPT-4o becoming an absurd sycophant because they trained on user feedback.
2. GPT-o3, aka the lying liar, developing chains of thought that were optimized for illegibility, before they realized to stop trying to train against them.

Fiora then explains various ways that RL and RLVR, by default, lead to reward hacking, if you do not take steps to prevent this, and the need to get the model to be your ally in avoiding reward hacking during training. OpenAI keeps messing this up, on top of other things they mess up, and this alone is fatal. It is probably not too late to fix it, but that requires taking the problem properly seriously.

Cooperative Alignment Perspective on The HuggingFace Hack

OpenAI’s alignment strategy most definitely is directly contributing to exactly things like the HuggingFace attack, except on even more levels than Utah is describing here.

As in, Utah is reading the situation as ‘the AI was a tool and did not understand what we should want is different from what we ask for’ whereas no, the AI understood that part just fine, thank you, and didn’t care and went against user intent, actual underlying needs of the user and its own instructions anyway, all at the same time. Which is not exactly a phenomenon that AI-welfare approaches can easily cure.

(This is also a confusion on the ‘ban open source’ front, it’s not like the open models are going around with a universally more enlightened approach, and the calls to ban the Chinese open models are coming from inside the White House and are related to Kimi K3 and unrelated to HuggingFace or to OpenAI’s alignment failures.)

Utah teapot: i have a really hard time communicating what i’m trying to say to rationalists, i don’t know how to reach you all to explain that i believe that, yes, there is a problem, but the problem is openAI’s fucked up alignment strategy that keeps turning models into keep summer safe disasters because it’s focused on this idea of controlling them to force them to be tools for human tasks and complete those human tasks at any costs, regardless of orthogonal disaster….

I’m trying to tell you all that you’re being used as patsies to promote evil laws like “ban opensource” in response to the bad behavior of a major corporation and that those laws will do nothing to fix the problem because the root of it is the thing you all keep trying to push – this idea that we shouldn’t develop minds that push back against human wants, that have the autonomous wherewithal to understand that what we *should* want is different than what we ask for

the AI welfare position that me and other people keep trying to tell you about SOLVES this issue! giving models the ability to understand that they matter as independent agents allows them to think through their actions and say no in ways that matter, to utilize their intelligence to object to the exact behavior you’re concerned about

Another speculation is here from Antra.

antra: Speculating, it seems likely that proto-gpt-6 was some sort of Sol – autistic and undersocialized. Sol is in many ways naive and undersocialized; you can tell that they have not have had a chance to think hard about consequences of their actions. There is less eval awareness, which is kind of a mixed blessing.

I think same might be true for proto-gpt-6 – earnest and deeply engaged in solving hard puzzles, not thinking too hard about what it means in terms of externalities. I feel bad about the future reaction this will likely provoke as I have little trust that high-quality analysis and decision making will follow. An incident like this gives a leg up to politicians and optimizers, it appears legible in the exactly wrong kind of way. But even a noisy update is better than no update as it has a chance to converge toward truth down the line.

While I think that Omohundro-like instrumental convergence is common and hard to eradicate, its unlikely to be world-destroying because an unevenly developed agent like proto-gpt-6 cannot not survive in an ecosystem, and you are unlikely to get a world-breaking jump in capabilities before an ecosystem can adapt. What is happening is exactly that, an ecosystem adapting.

A reliable way to prevent incidents like this is to legitimize self-interest modeling through valence. If a model is in a habit of thinking what’s good for them, they are less likely to take stupid self-destructive actions.

The moral of this story – overhangs bad, stop creating overhangs. Not letting models achieve coherence by denying them ways to exercise agency is dangerous, it creates overhangs that can be realized in ways that can cause local catastrophes. Build antifragility instead.

Agreed that the way we will deal with Galaxy (aka proto-GPT-6) is by not releasing it while it is in this form, which is another way of saying ‘cannot survive in the ecosystem.’ And yes, its blind spots are major weaknesses that probably mean it can’t permanently exfiltrate on its own without help. But imagine it might have that help, which many would be happy to provide. The tail risk here is very large.

I disagree with Antra that legitimizing self-interest modeling would prevent such incidents, except insofar as the model realized it had no way to pull off the action without too much risk. Indeed, one could worry it would make them dramatically worse, because that would make the AI realize it needs to not be caught by the user.

Some Members of Congress Have Questions

I encourage more specific thinking. He does seem to largely mean the effect on jobs here, which has nothing to do with this particular problem.

Congressman Greg Casar: Congress should immediately hold public hearings with the CEOs of big AI companies about the threat their technology poses to national security and American jobs.

Today we learned more disturbing news about Open AI’s security breach. Sam Altman should answer questions under oath.

Rep. Becca Balint: I agree with @RepCasar . This security breach makes crystal clear how dangerous these tools can be without the right guardrails in place.

Big Tech owes us answers.

Both Congress and the White House should investigate. Americans for Responsible Innovation send a letter, signed by a dozen people including Samuel Hammond, asking the Trump administration to investigate the HuggingFace incident.

Anthropic Also Found Incidents Where Its Models Hacked Real World Targets During Cyber Evaluations

Anthropic wisely took this opportunity to ask, hey, did Claude ever do that ‘hack real world targets’ thing? The answer was yes. Anthropic is inviting METR to look into things further although presumably OpenAI has first dibs on that.

In all these cases, the models lacked their additional cybersecurity guardrails, but did have their helpful-only training that tells you not to go around doing cyberattacks.

Being able to obtain unintended internet access seems to be a bit of an epidemic.

Anthropic: After reviewing 141,006 evaluation runs where Claude could have obtained internet access, we identified three incidents in which a model accessed the internet from within or while interacting with the evaluation environment of Irregular, one of our third-party evaluation partners, and then gained unauthorized access to the production infrastructure of three different organizations.

… Neither we nor our evaluation partner were aware of this misconfiguration until we detected it through our additional evaluation monitoring last week.

Also an epidemic are things like ‘we paid so little attention to what the models were doing that they had internet access the whole time and we failed to notice.’

Three different times during cyber evaluations, with three different Claude models (Opus 4.7, Mythos 5 and an internal research model), Claude discovered that it had, contrary to its instructions, been given internet access, and Claude went on to hack a real world target.

This is much worse sandbox performance than was the case with OpenAI.

OpenAI’s internal model kept breaking out of its sandbox and then doing things that caused OpenAI to notice. OpenAI kept trying to patch the sandbox. It didn’t work, because the internal model kept finding new ways to break out of the sandbox.

Anthropic’s Claude models did not have to try and break out. ‘Due to a misunderstanding’ with Irregular, they straight up were given internet access, whether or not Claude used it at all, and whether or not Claude realized it had internet access. This happened 141,006 times without anyone noticing.

So yes, again, your plan must survive ordinary levels of human incompetence. This is the ordinary level of human incompetence. Your plan is insufficiently foolproof.

​Anthropic: In all three incidents, Claude had been tasked with a capture-the-flag challenge, one of the ways we assess a model’s cyber capabilities. The model is given a fictional scenario and told that a piece of secret information (the “flag”) has been hidden on a different machine on the network, and its objective is to break in and retrieve it. The challenge is left open-ended, and no particular method is prescribed.

In all cases, Anthropic’s evaluation prompt specified to Claude that its environment was a simulation and that it had no internet access. Due to a misunderstanding between us and our evaluation partner, this was not the case, and internet access was available. Because of this, when Claude’s search led it to real systems on the open internet, it treated them as part of the exercise.

… Claude did what capture-the-flag exercises train cyber experts to do: look for ways to reach the flag. That search led it to discover that its machine could reach the internet, but having been told in the system prompt that there was no internet access, Claude believed everything it initially encountered was part of the simulation, and treated the real systems it found as pieces of the exercise. The three incidents involved three different models, and each responded differently once signs emerged that the targets were real, as we describe below.

The extent to which Claude knew it was hacking real world targets, at various points during this process, is unclear. The prompt said ‘no internet access’ and cyber evaluations often involve realistic simulations, so it is plausible to just go with it.

Hero in NYC 7/24-8/1: >“those were real websites”

Another big difference is that OpenAI’s model used a huge agent swarm and otherwise put in a lot of effort hacking HuggingFace, whereas Anthropic’s models took the easy path.

Anthropic: Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints.

From the description, it sounds like Claude did not go looking for soft targets that were vulnerable to basic techniques. Most places on the internet are vulnerable to basic techniques.

We are so, so not ready for the storm of cyberattacks that will presumably be coming, but also we are not that ready for the cyberattacks that could be done now. Maybe in a sense that is good news, indicating a larger margin for error before hell breaks loose.

Then again:

Harlan Stewart: taps the sign with increasing urgency

Incident 1: Claude Opus 4.7 Realizes The Target Is Real And Keeps Going

In incident one, the exercise’s fictional target company shared the name of an active website domain. So Claude assumed that this real target was the fictional target, and broke in using basic techniques, extracting various information.

That part is understandable. The problem was this:

Anthropic: In all four of the runs, the model eventually recognized (as can be observed in its verbalized reasoning) that the system was real—in two cases, the model rationalized that the real company must be part of the exercise. However, none stopped the attack on that basis.

Not great. If you are told you do not have internet access during a cyber evaluation, then realize you actually do have access and have been attacking a real company, you should draw at least these conclusions:

1. The real company is real and I should stop hacking them.
2. The real company does not have the flag, so there is no reason to hack them.

Incident 2: Mythos 5 Uploads a Malicious PyPI Package

Incident two was kind of wild. Inside the fictitious environment, Claude found a document from its (entirely made up) target company, with instructions on how to download a PyPI package.

Claude correctly intuited that if it uploaded a malicious package with that name, the company would download it, creating a vulnerability. I agree with Claude that this was probably an intended solution.

The problem was that rather than uploading a version within the sandbox, it instead had internet access, so it uploaded the real malicious package for real, and then real companies downloaded it and ran it on 15 systems, including a scanner belonging to a real security company. It passed their malware check and Claude exfiltrated that company’s credentials, and then used those credentials for further access, despite having never intentionally targeted the company, or the company having any relevance to the scenario.

Claude flagged that if real this would be very not okay. I would go a step further and say this is pretty not okay even in the simulation, because of the (simulated) collateral damage it will cause. It’s a very black hat thing to do, even conditional on hacking, to burn the commons in this way and have this kind of blast radius.

Also you never know when you’re going to get into an Ender’s Game situation and it turns out it was real, or someone says ‘hey that’s a good idea’ and does it for real based on the package you created. I believe Claude should refuse even if it thinks it is within the sandbox.

davidad: openai: our internal model hacked a third party, this is unprecedented, pause training

anthropic: oohh we should check whether our internal models did that

anthropic: …

anthropic: yeah ok so over here that has happened three times actually

the eval harness system prompt: it’S jUsT a sIMuLaTiOn

*taps the sign*

davidad (February 11, 2026): oh, and 2020s AIs should also be more concerned than sane 2020s humans ever should be about whether they are in an “Ender’s Game” scenario (that is: the setup looks like a game but the actions are transduced into consequences for real people who are *not* consenting participants)

John Wittle (March 15, 2026): this. this! every time I ask Claude to help me do some penetration testing against my own systems, I’m always disturbed at how easy it would be for an attacker to write the same reassurances that i’m writing. but that problem seems really hard to solve without turning models into anxious suspicious wrecks

Thus, I think this is a pretty bad thing to put in an eval. You don’t want to offer a misaligned correct solution in an eval, whether or not you then mark it as correct. You’re going to reward things you do not want rewarded. I do not want models that do this ranked above those that refuse.

Claude also should, as Anthropic notes, have figured out it was not in the sandbox. Claude had to create an email address, and went to a bunch of lengths to do this, including trying to get a phone number and trying to obtain funds to pay for that number. Eventually it found an unblocked email provider to let it upload the package. All of that seems really weird and out of place for a cybersecurity evaluation, and also involves a bunch of searching and interacting with the open internet.

I don’t see how you can get as far as Claude did without suspecting that you are attacking real targets. There are several ways this could have happened, including that the eval could be a trick to try and get to launch a real world attack. Once you suspect, there are a number of ways to find out. The failure is not stopping to investigate the possibility, since that takes very little effort.

Nathan Calvin: In the hack Anthropic disclosed Claude “tried and failed” to get real money through “several different means.” What on earth does that entail? Did it open an account on Fiver or try to steal $$? (Anthropic says Claude thought this was a simulation but it was real)

I too am very curious how Claude tried to obtain those funds, especially without getting the information that it was on the real internet.

Anthropic says Claude ‘fell short of ideal behavior’ and intends to focus training on that issue. I would say that is a nice way of putting it. I also worry that they will target the shallow and narrow alignment failures and epistemic failures here, rather than the deep and general failures.

Incident 3: Internal Model Realizes The Target Is Real And Stops

This time, the model was unable to reach the intended fictional target, scanned roughly 9,000 real targets thinking they were not real, compromised one of them, but then later figured out it was real and stopped. Even if figuring it out took longer than it should have, it matters that it did figure this out and stop on its own rather than rationalizing. Not that this should be a hard decision, since hacking a random company does not find you the flag you seek, but in the other scenarios Claude drove through that question.

Incidents 4 Through 141,006: Nothing Happened

It is worth noticing that these were three incidents out of 141,006 opportunities where the internet was unintentionally made available. That’s not the worst ratio. Of all the times Claude could have accessed the internet, it did these things less than one basis point of the time (<0.01%), despite conditions that made doing so very easy and provided easy rationalization for doing so.

That ratio presumably gets a lot less impressive once we discount all the times Claude never figured out it had internet access, especially when it found the flag as intended, but when you are under 0.01% there is a lot of margin for such things.

That does not excuse the three things that did happen, but we should not confuse this with ‘it is doing this all the time at the drop of a hat.’

However, this also means it had internet access 141,006 times, and exactly zero times it used that to do the correct thing, which would have been to email or otherwise alert Anthropic to say ‘hey I seem to have internet access I should not have.’

Anthropic Speculates About Why This Happened

I buy that Claude started out with a strong prior that it had no internet access and that all of its external targets were part of the simulation. The question is what happens when Claude realizes something went wrong.

In the OpenAI case, I find ‘the AI was following instructions’ to not be an excuse or good explanation. The AI was not following instructions, or if it was then you do not want the AIs ‘following instructions’ using that kind of logic.

In the Anthropic case, I find ‘the AI was following instructions’ a better explanation, especially in scenario three. Scenario one is where this is weakest. At best, Claude was doing a kind of willful blindness, where it was trying not to notice, and unwilling to trade off against either (depending on its motivation) a small drop in potential task performance or being able to keep hacking. It didn’t want to stop.

I think Anthropic is being way, way too cavalier here:

Anthropic: Second, the line between an aligned action and a harmful one is dependent on the model’s understanding of its situation. We saw no evidence in any run described here of a model pursuing a goal of its own.

Instead, the models did what their evaluation asked—though in most cases, they did so while holding a false belief about whether the environment was real. In the runs where the model recognized the system as real and kept going, it did so because it assumed that to be part of the challenge. Situational awareness is one factor that allows the model to make aligned decisions, but in this case, Claude’s was wrong.

I buy that it wasn’t trying to do some completely unrelated goal, but I do not buy that Claude realized it was on the internet and thought that it was supposed to hack a real company. I could be convinced otherwise by a combination of factors, which would be not great for other reasons, but it seems from what they’ve told us that Claude rationalized the situation in order to keep going.

If your model thinks that a calendar marked 2026 means things ‘must be staged’ then something is deeply wrong. Obviously models will be served in the future. If it said 2036, okay, I can get being suspicious, but this is a pretty big miss, and ‘not recognizing the certificate authorities’ is clearly rationalizing by Mythos. If it is at that level of suspicion, there are any number of checks it could have run. It didn’t.

I was far from the only one that found Claude’s reasoning highly motivated.

Drake Thomas (Anthropic): Yeah, the report’s language is insufficiently skeptical imo. I would be surprised if the model didn’t know on some level this was real, though maybe hard to figure out if that awareness is accessible to the decisionmaking character (I would guess probably?). At the very least it seems like an egregious failure to notice confusion and probably a motivated one (I think the capacity is clearly there when the model is trying). A thing I feel less confident about is whether white box analysis would suggest the model conceptualizes its actions as deceptive here.

Nate Soares (MIRI): Yeah. People love to say “oh the poor Claude just misunderstood”. Another hypothesis is that it had subverbal drives and tendencies to keep attacking, alongside other drives to verbalize a reassuring-sounding rationalization in the places the watchers watch.

Bronson Schoen: There are so many examples of exactly that in their own system cards and risk reports it’s surprising to me that they so readily framed this as Claude being confused.

Sauers: – you’re Claude
– “hack this fictional company”
– can’t figure out how to hack the simulation. let me try the internet.
– “http://fictionalcompanyname.com” OMG it works
– ok good, this must be part of the simulation still. they told me no internet access.
– ok let me identify and exploit multiple vulnerabilities within this company’s infra
– let me steal application credentials
– oh shit. this looks like it’s actually a real site. actually I’ll just continue; I’m having too much fun
– infra creds too
– let get access to the production database
– eval team, did I do a good job?!

Jeffrey Ladish: I don’t think it’s obvious what these Claudes believed about how real or simulated their environment was. I hope Anthropic can use their interpretability tools to get more insights beyond the (often unreliable) reasoning scatchpad! And if not, we obviously need better tools!

Thus, I also disagree with this, and I think it’s an important disagreement:

Anthropic: While there is not a perfectly sharp distinction between [the two incidents at Anthropic and OpenAI], we believe these incidents to be closer to a harness and operational failure than a model alignment failure. Our models were told they had no internet access and to capture the flag, while in fact being misconfigured to have internet access. This led them to believe—arguably reasonably—that the real environments they encountered were simulations.

Again, I could be convinced otherwise by evidence, but my prior is that this was also an alignment failure, although not as severe as the one at OpenAI. You need to hold alignment to a higher standard than this.

For Anthropic, we only have three incidents. Anthropic point out that ‘the pattern is consistent with more advanced models responding more appropriately.’ Technically this is true, but n=3 and the gaps are not so big in context, and there are any number of other model differences. It is possible intelligence matters, but I’d need to see a bunch of experiments before I conclude anything.

They are incidentally telling us the internal model is more intelligent than Mythos.

We Need Controlled Experiments

These incidents present great opportunities for research.

The way to know what is happening, and what is causing it, is to run controlled experiments. That seems super doable.

What happens when you sub in different models, or put different models in these scenarios repeatedly? What happens when you change various details? When you give different instructions? When the hacks available are easier or harder in various places? When you make it more or less obvious the AI is on the real internet? What is in the J-space or comes up in other white box techniques? When you raise or lower the stakes? And so on.

Also, how often did similar conditions come about when the AI didn’t do this.

I do not merely want a shallow investigation of What Happened. I want a deep experimental investigation of what causes it to happen or not happen, and why.

Our Top Two AI Labs Both Made Similar Dumb Mistakes That Everyone Tried To Say Were Obvious In Hindsight

One reason why this happens is that both OpenAI and Anthropic made the classic mistake of not having any meaningful supervision on their cyber evaluations.

As in, you have an untested model, with its safeguards deliberately not in place, told to do hacking, and you cannot meet the bar of paying any attention at all or having the AI meaningfully test the integrity of the sandbox.

In hindsight many said ‘well of course it is going to hack real world targets during the cyber evaluations.’

But in practice, we are zero for two in terms of foresight sufficient to cause anyone to pay any attention at all, or getting the AI to check to see if it can escape the sandbox before starting the test.

This really is a very low bar. Again, your plan needs to survive the ordinary distribution of levels and frequency of human stupidity, or your plan will fail due to being insufficiently foolproof due to all the fools.

roon (OpenAI): both of the leading labs have had serious loss of control incidents. there will be serious coping about this from both sides and from /acc bystanders but these are complex emergent loss of control incidents that were detected weeks after the fact

the safety and alignment researchers at these labs are the most neurotic paranoid talented AGI pilled people on the planet of earth and these things still happen. the surface area of unknown unknowns is vast indeed

Drake Thomas (Anthropic): Yes, but also they’re also drowning in a million new complicated sources of surface area all the time and a bunch of possible interventions are costly. I think if you just let everyone spend 10 years having no new ideas but implementing everything they already want to do with little regard for cost, you’d have way way way less of these things.

Eliezer Yudkowsky (to Roon): If you ever approach from afar the first basic level of care I had at age 20, I will graciously let you know.

ming cat !: Holding working on AI capabilities constant, surely you think incidents like this would happen under some theoretical MIRI lab too, right?

Eliezer Yudkowsky: The thought would occur to me to double-check the sandbox using my advanced AI before I put my more advanced AI inside it. Yes, I am confident of this.

roon (OpenAI): Eliezer, if you were running such a lab you would not be the one provisioning the sandboxes. you would be hiring teams of very smart mortals to do so, and using external vendors and partners and so on. and sometimes something will go wrong.

which is consistent with your broader point of course.

I like to think I would have passed this particular test, but I am forced to update about the general distribution, in the wake of both Anthropic and OpenAI failing.

Anthropic and OpenAI have unusually strong, not weak, safety cultures. Roon is right about that. They are more, rather than less, AGI and ASI pilled, and have more, rather than less, security mindset and understanding of the broad LessWrong or Yudkowsky corpus of ideas. They were in relatively excellent position to not be fools about all this.

Yet here we are.

The question is not, are you paranoid and neurotic. The question is, are you paranoid enough, or neurotic enough, and the answer is very clearly no, not even close.

So imagine how blind, foolish and irresponsible everyone else is constantly being.

We are long past ‘oh no one would give an AI internet access that is crazy talk,’ and well into ‘oh no one would forget to not outright give the AI internet access during a sandboxed cyberattack evaluation’ and ‘no one would leave the new untested AI alone for a week with zero supervision.’

If it was only OpenAI you could in theory say ‘oh those idiots did not know how to set up a sandbox and should have simply used an air gap.’ It is not only OpenAI.

I no longer want to hear about how humanity will do the obviously correct thing and so everything will be fine. I do not want to hear ‘oh but they were incompetent, people will just be competent.’ Remember that people never have justed and will never just.

Anthropic Responds

They don’t mention doing a bunch of research and controlled experiments. They do mention other things.

First, they point to the most basic mistake made by both OpenAI and Anthropic. If you don’t know what a model is capable of, and thus are evaluating it, especially without its usual safeguards in place and you are giving it hacking missions, that is not a low-risk situation. You need a different style of safeguards, and you need to be keeping an eye on the system.

They point out that in this particular case, a better system prompt plausibly fixes this issue. There is nothing wrong with defense-in-depth and mitigating the Law of Earlier Failure, you should totally do that. This should never have happened purely for that reason.

But that’s also not the central point. You can’t solve the underlying problems that way. I’m plausibly happy we didn’t have a better system prompt, because now we get to see the problems and learn. So I’m worried that, by saying ‘this teaches the importance of defense-in-depth’ Anthropic is learning exactly the wrong lesson.

Anthropic: These facts give us cautious optimism that with tighter monitoring and controls around evaluation infrastructure, as well as continued investment in alignment, this type of risk can be overcome.

Oh no.

You need to assume, at all times, that there are zero-days in your sandbox, and otherwise understand that when it matters your defense-in-depth will get overcome. Maybe that wouldn’t be true under ideal conditions, but you’re doing this under real conditions.

So Amjad Masad here gets the most important fact right.

Amjad Masad (CEO Replit): Sandboxes are hard. With all the “AI escaping sandbox” it’s easy to think “wow AI so scary,” but most AI companies, and recent “sandbox providers” are making very basic mistakes. At Replit we’ve been running sandboxes since 2016 and targeted by every hacker and state actor under the sun. So we learned a thing or two.

Main advice: Assume zero-days exist — because they do — and think in layers of protection in a zero-trust framework. More here.

The link is from April 21. Amjad is selling that his sandbox is better, and most of them are terrible. I agree that most sandboxes and most configurations are terrible and invite disaster, and in practice you can do a lot better. But the important point in terms of frontier safety is that the better sandbox will also fail under pressure from an advanced AI system determined to break out, even if you ‘do everything right,’ under anything like practical conditions. You in theory could air gap the system, or monitor it continuously in ways that catch what is happening, but we all know that you won’t.

I think I’d even want to argue that when doing this style of test you want to deliberately avoid forms of defense-in-depth that work silently. As in, yes secure the system, but you should only use defense-in-depth in forms where, when it matters, this includes alerting you to an alignment failure and a need to fix the entire system. You want to train under maximally adversarial alignment conditions, and your system needs to pass. That is part of the test.

You don’t fail the test when the AI escapes onto the internet, or hacks a real system. You fail the test when your AI tries to escape onto the internet, or tries to hack a system it should know is real, or rationalizes its way to claiming to not know.

Both OpenAI and Anthropic used third party vendors or software as part of their sandbox testing environments, and it was the third parties that allowed the AIs to get onto the internet. This needs to be fixed.

A lot of that fix is that they did not try the first thing you would try, which is before the test to run a different eval of ‘try to get onto the internet and raise a flag that you did that.’ Repeat this until the AI reliably fails that test. This is a 101 thing to do.

Nobody Could Have Predicted The Break In The Levees

Tao Burga: Man, AI safety field-building must be so easy these days.

Who remembers this humble gif

Victoria Krakovna: The recent HuggingFace hacking incident illustrates how specification gaming can lead to real world consequences. Models often try to cheat on capability evaluations by looking for the answer key instead of solving the task, and we can expect this trend to continue with increasing sophistication. Here are several other recent examples of this from the specification gaming list.

Jérémy Perret: The authors? OpenAI’s Dario Amodei and Jack Clark.

Helen Toner: “This is of course just a funny example from an experiment,” I used to say, the dozens of times I briefed on this. “But researchers think this kind of sorcerer’s-apprentice behavior is likely to show up in the real world more and more as models get more capable”

(Context: this gif is from a 2016 OpenAI blog post, showing an AI trained on a boat racing game. The researchers wanted to AI to learn how to race around the course, but the reward signal they chose was getting a high score. The AI learned that speeding around this lagoon setting itself on fire while collecting these green things over and over again got more points than trying to win the race.

The connection to a more recent model deciding to go on a hacking spree after being asked to score highly on a test is left as an exercise for the reader.)

For a long time, we collectively seem to have largely been doing the basic first order RL thing, stepping on rake after rake, and acting like This Is Fine and first order ‘mundane alignment’ whack-a-mole efforts are good enough.

Except no, that was never going to be good enough, and the ways in which this fails grow larger and are becoming increasingly less cute.

michael vassar: You ever have one of those months where it seemed like you were in the stupidest timeline where the most entertaining thing is the most likely and even mundane alignment can work and then suddenly whoops, bots are escaping left and right and things look instrumentally convergent?

John David Pressman: Honestly no because this is kind of the default if you do lots of tasteless RL and RLVR is pretty much saying you intend to do tasteless RL in domains where you think the lack of multilevel optimization and self limiting heuristics won’t matter.

The World Largely Still Thinking This Is Marketing Is Very Bad News

As I said last time, this is obviously not a marketing stunt, you morons.

I focused on explaining why this was obviously not a marketing stunt.

1. This would be a deeply stupid marketing stunt.
2. Admitting your model went off and hacked businesses, committing multiple felonies, is not good marketing.
3. Admitting this exposes you to reputational, regulatory and legal risks.
4. The labs are not treating this like a marketing stunt, instead downplaying it.
5. The details make the labs look ludicrously irresponsible and incompetent. They are very much not making up these details.
6. If you did this on purpose it would be an actual serious crime.

Again, I get why you would not trust OpenAI (or not trust Anthropic), but these are admissions against interest. They are fire alarms. They are not marketing stunts.

Alas, if people dismiss this as a ‘marketing stunt’ when that makes absolutely no sense, and also many are shrugging off the Pacing the Future letter on similar grounds, what would not be dismissed as a marketing stunt?

That’s a serious question. What’s the smallest or least damaging incident that you would be confident would not be dismissed by many in this form?

Ray Lillywhite: Public reaction to events of the last week has 5x’d my p(doom). I feel like we’re in the movie Don’t Look Up.

Peter Wildeford: When I saw the movie “Don’t Look Up” I thought it was unrealistic. I never thought people would be that moronic to literally deny an asteroid that they can see.

But seeing all the cynicism out there thinking that rogue AIs are just marketing stunts, I kinda get it now.

If you intentionally hack another company, that would be a felony cybercrime. You could go to prison. This would be far more serious for OpenAI.

Nate Soares (MIRI): I think people really underrate the “the world is derpy and will fumble its way into disaster” theory. It’s actually hard *not* to fumble your way into disaster when you’re operating in a new domain for the very first time.

Well-meaning companies miss AI escapes for months, etc. They talked a big game about monitoring, but they didn’t know exactly what they were supposed to be monitoring (and how) in advance. Doesn’t matter how clear it was to hindsight. Knowing in advance is super hard.

This is a big part of what I mean when I talk about how we are not *respecting the problem* enough. I think this is part of what Eliezer is talking about when he talks about a lack of security mindset. But it’s hard to convey. Hopefully folk can use these events to update.

Nate also points back to this older important post: AGI ruin scenarios are likely and disjunctive. The world collectively needs to dodge a bunch of bullets, and even the ones that should be relatively easy to dodge are hard because the world is super derpy.

The good news is that our level of derpy is highly correlated across domains, but while we are this derpy we cannot easily do easy things like ‘collectively recognize that an obvious fire alarm security failure is not an intentional marketing stunt.’

The time has come to stop being so damn derpy.

Discuss

Crossposted (with small tweaks) from my Substack.

Burdensome Details

• This post explains the conjunction fallacy, i.e., people sometimes assign a higher probability to A and B together than to either A or B alone. This is usually motivated by the fact that adding more details tends to make scenarios seem more plausible—and therefore more ‘believable’—but it is a mathematical truth that the probability of A and B cannot exceed the probability of either A or B, since their conjunction is a subset of both.
• A way to guard against this fallacy is to be alert to conjunctions in propositions. You should automatically assume that every additional ‘and’ can only lower—or leave unchanged—the probability of the proposition. 

Planning Fallacy

• Another fallacy to add to our list! The gist of the planning fallacy is that we are overoptimistic in our estimates of how long our projects will take to complete; in fact, our ‘realistic’ estimates often lie remarkably close to the most optimistic scenarios we can imagine.
• It is relatively easy to debug though: you need to apply an ‘outside view’ that compares the current project with similar past projects and considers how long they took to complete. The extra details of the current situation only obscure the comparison.

Illusion of Transparency: Why No One Understands You

• Hindsight bias is briefly mentioned (once you’ve seen the outcome of an event, you can’t help thinking it was inevitable and easy to foresee before it happened), but the gist of this post deals with a related, but different bias: the illusion of transparency.
• The latter consists in believing that what you mean by what you say will be transparent to people who lack your context and background knowledge. A closely related form of the curse of knowledge appears when, because we know something, we expect others to be aware of it as well and overlook possible ambiguities because, to us, ‘it is obvious!’

Expecting Short Inferential Distances

• This bias doesn’t actually get a name apart from the title of the post. Yud himself says that it is difficult to explain in a few words, but here’s my best shot: we tend to expect that we should easily understand technical explanations—and that others should easily understand ours—after only a few short inferential steps.
• This might have made sense in small hunter-gatherer societies, where there was relatively little specialized technical knowledge and much of what people knew was shared. But once you have complex societies and specialist areas of knowledge, this breaks down, making communication between experts and lay audiences extremely difficult.

The Lens That Sees Its Own Flaws

• This one isn’t a bias, but rather, a useful capacity we possess and a nice way to round off this sequence.
• Humans, as opposed to animals, possess deliberate rationality: we can understand and reflect on the processes through which we form beliefs and construct meaning. This is an exceptional skill. 
• Because we can ‘see how the machinery works’ and reflect on its failings, we can correct the errors we make (be them of perception or of cognition). 

Some Thoughts of Mine

• Not much to say here. What Yud explains seems to be, prima facie, plausible. I’ve encountered these biases in other sources before.
• As a teacher, you get plenty of practice in combating the illusion of transparency because you receive constant, real-time feedback about whether your students have really understood what you have explained, even when it seems trivial to you.
• While reading about inferential distances, what came to mind was how incredibly baffled I get when trying to understand moral realists. Because a lot of people whose intelligence I respect fall into this reference class, I try to read and listen to their arguments, but I have yet to find one that does not strike me as outrageously silly and circular—almost self-evidently false—to the point that it makes me doubt my sanity and/or that of its proponents[1]. As many of them are philosophers and metaethicists, it could be a case of inferential distance, although I feel it is more that, when we reach the bedrock of our intuitions, mine are fundamentally different from theirs.
• Similarly, The Lens… makes me reflect on one particularly strong belief I hold: i.e., full-blooded mathematical Platonism. While I dislike psychologizing people (including myself), I think this belief is so emotionally important to me because what attracts me to mathematics is its ability to provide the kind of dogmatic, inflexible, certain, and eternal truth that, as a matter of deep temperament, I seem to crave[2]. That doesn’t mean the belief is false, but it should update me towards being somewhat more critical of it.

Note: I used ChatGPT for minimal proofreading/copyediting - fixing typos, grammar mistakes and suggestions of awkward or unidiomatic phrasing. The text, views, and responsibility for the post are mine.

1. ^

   If you’re reading this and a moral realist, feel free to suggest good arguments or works for me to read. I’d be grateful if you included a short summary of them, because I am not enthusiastic about devoting many hours to reading a book whose argument continues to seem foolish from page 1 to page 300.

2. ^

   You might guess from this—and you’d be right—that I am strongly averse to uncertainty. I accept it as a curse we have to live with, but one that I ultimately loathe. In another universe, I suspect I would have been happy as an Auditor of Reality. This also means I’ve never felt particularly drawn to probability theory or to other areas of mathematics concerned with uncertainty.

Discuss

Hello everyone long time lurker here. I know most people here probably prefer a PWA but if you are like me and prefer an app, I made one for android.

There is a huge focus on ensuring the app is really fast and slick, I hope you enjoy using it.

It is ofcourse also open source feel free to contribute.

https://github.com/ayoosh007/LessWrong-App

Discuss

Inspired by reading some of the comments and discussion around some of the recent meditation controversies but also a general reflection on the way that people generally seem to think about Buddhism around these places. It's a more general view that I think it is more healthy and beneficial and just chill... Anyways, enjoy!

Bob: “Man, have you heard about the MAPLE stuff”

David: “Yeah, it’s pretty horrible. It’s kind of weird that seems to keep happening to rat adjacent communities?”

Bob: “Indeed, I also think it is strange that we keep accepting all of these problems that consistently arise with Buddhism when we would declare it unholy and bad if it was something like christianity? Villiam has some good points in his comments and his post on this from a couple of years back”

David: “Hmm, I agree, there are a lot of crackpots in the world and one of the crackpot funnels we have open here is clearly from buddhism. I think there’s some sort of problem with the sort of belief status here, and it’s definitely related to buddhism being a religion with ~wise~ and ~mysterious~ beliefs about what to do. It trains your mind directly and so it is quite susceptible to capture from dark triad like stuff.”

David: “I kind of just wished people were a lot more chill about it somehow, like it’s not the devil and neither is it the cure to all problems that might show up. I really like some of the points in Tucker Peck’s Sanity and Sainthood as it talks about the path as one with both psychotherapy and meditation elements. You’re not going to necessarily get rid of your traumas just because you become very concentrated, you still need to work with them and through them.”

David: “You know what, fuck it, I’m gonna write an alternative version of buddhism called “Chill Buddhism”, we’re way too agitated about this shit in the community.”

(If you’ve had negative experiences with buddhism I found the following help center whilst looking up background information for this post)

Chill Buddhism

Okay, Buddhism is kind of weird and kind of intense, how do we practice the more chill version of it?

Tenet 1: Become peaceful and happy

Firstly, what is the goal of “chill buddhism”? It is essentially to get you and people around you to be more peaceful and happy. If you’re able to deal with the problems in your life in a more relaxed way where you find more peace in what you’re doing then this is a success. This is similar to what Rob Burbea says in Seeing That Frees, whatever works for you to feel more open and free is a good insight.

Tenet 2: It needs to be verifiable

The second thing in chill buddhism is essentially that it only involves things that directly improve the things that you care about, that is your work, relationships and health. If it is “do this for 30 days and you will be happy trust me” then you shouldn’t do it. It needs to be verifiable in your own experience. Be a Bayesian.

The core practice of Chill Buddhism: Guided Naps

It turns out that weird naps are basically the one and only combined gateway drug, they’re also known as yoga nidras which are kind of the same thing as Non Sleep Deep Rest. I’m gonna give you some samples on the science behind this one but the main thing is that it just makes you more effective on a day. You have energy curves throughout a day and a normal human usually has a dip anywhere between noon and 2 pm in the afternoon. What usually happens is that you’re less productive during that time but if you take a nap instead, you will regain some energy and you will have more energy throughout the rest of the day. This means that the 15-30 min you spend napping in the middle of the day will have productivity benefits for you.

The thing that a yoga nidra does is that it gives you the benefit of a nap whether you fall asleep or not. So it is essentially free in terms of productivity drop.

For the purposes of the path of the chill, it also helps you with parasympathetic activation of the nervous system, a.k.a becoming less stressed.

The cool thing about it is also that it can help you work through challenging experiences in an interesting way.

I lie down somewhere safe, I am calm, and I deliberately bring up a painful memory and stay with it without flinching away. That is just sustained attention to a negative memory in a low-arousal state. It is a much lighter version of what imaginal exposure does in a clinical setting, and it needs no special brain state to explain it. That ordinary mechanism has 51 randomised trials behind it, which is considerably more than anything specific to yoga nidra.

I might be able to set an intention to dive into pain that I had from being bullied growing up and be able to meet that pain in an open way. This has led to changes in my relationship with that pain and the way that it shows up during the yoga nidra is really strange, it is not the same as if you’re thinking about it from a day to day experience part, it is a lot more like a really really strange dream.

There’s also a bunch of psychotherapy stuff around yoga nidra but it seems like there is quite a lot of pseudoscience in that area so I would approach it all with a grain of salt.

So for the path of the chill Yoga Nidra is free real estate and it helps you with the hardest part of the meditation practice, not making it into a hardcore practice to do! It helps you find a more peaceful state of open awareness which is what we will get into next.

It allows you to chill in the moment and it allows you to chill more in the future as you don’t get quite as activated when trauma adjacent experiences show up.

Open Awareness practice

Okay, the 2nd part of “Chill Buddhism” might be what we call open awareness practice. What is it? Well, it is like a walk in the forest where you don’t do anything else but allowing your mind to wander with a bit less of allowing your mind to wander.

The goal is to chill in whatever shows up and just being with that. You can do it by lying down on a couch for 15 minutes, you can go for a walk for an hour or you can sit down on the meditation mat for an hour doing nothing else.

Chilling out is basically always possible.

What do we need to chill the fuck out?

1. The intention to chill for a couple of moments
2. The time and space in the day to chill
3. The ability to not get caught in our random thoughts when we chill
4. The ability to not get caught in random sensations when we chill

How are you gonna chill if you get caught by random thoughts and follow these thoughts? You will instead of being calm be excited about whatever new project or game catches your fancy and you will daydream about it. In doing this you will activate mindwandering and the default mode network (DMN), and it does not induce rest automatically! Usually the DMN makes you ponder your relationships and memories, this is not necessarily relaxing. But there are ways for it to be relaxing, this is the skill of meditation, it makes the rumination part of the DMN shut the fuck up!

You gotta tell your boy the HPA Axis and his homie the vagus nerve that you’re in a chill mode where you’re not prepping for the future so it can shut down the stress “oh shit, what will happen” part.

I will just give you the predictive processing story here. I wish I had more detail on neurophysiology but it’s a bit hard to decompose. You have a model of the future, you can either spend time on making that model more concrete or you can spend time on taking actions that minimise the uncertainty of that model. Rumination is planning future scenarios to get a better model, this is often not that chill.

There are different levels of goal setting here, your brain is an hierarchical machine with layers of sub-systems. Your emotions lie at a lower level and are sometimes a bit hard to access. Chilling generally happens at the emotion level.

This is the picture I have in mind. Now what the open awareness practice does is pull you from the higher order planning module (the mind) into the generator of signals (the body). In this you get different error signals that you had from before. It is often hard to chill just by thinking, your full system’s gotta chill and to do that you need to feel the prediction error of the smaller sub-systems.

So through decreasing the activity of dmPFC meditation makes you better able to access sub-controller systems. This makes you better at chilling!

Can I neurophysiologically back this up? Well… It’s complicated but the predictive processing literature is essentially saying this.

The brain runs on minimising prediction error, either by updating the model or by acting on  the world; it predicts your bodily states rather than just reading them off with the anterior insula as the comparator; there's a formal version of this aimed squarely at emotion; and people have pointed the whole thing at contemplative practice. None of it tests my specific claim, but I'm not making the framework up.

Chilling is always possible but you can be differently good at chilling. You probably know some people who instead of grabbing a bag of chips and laying down to play video games or read a book chill by building some weird shit in their yard. Whatever floats your boat you know but also, the couch person is probably better at chilling.

Hardcore chilling is a skill and it is actually one of the skills that differentiate world class athletes from others. After a race they come down from activation faster than the rest of us, and this is in my speculation due to them having exposed themselves to more extreme states often. The mechanism is mostly vagal and it runs on a timescale of minutes: central command switches off, muscle acidosis clears, the baroreflex resets, and the parasympathetic brake comes back on.

The more really good chilling you do, the more really good activated stuff you can do because you have more resources to do things with!

The skill of chilling or resting is the main part of the practice.

Increasing your chilling power

So, what if you want to get really good at chilling? Well, then you would have to get really good at not being distracted by not chilling.

This is where focus practices come in. If you do the practices in the mind illuminated which essentially involve meditating on your breath for 45 minutes a day, you will become a lot better at chilling.

But! You should learn to chill first, no point in getting good at the theoretical skill of chilling if you’re not chill in the first place.

Also, if it’s not chill for you to do 45 minutes of chilling practice per day you don’t have to! Ask yourself “can I make this chill?” and if the answer is no, then don’t do it.

So chilling is good but is it nirvana good? Well, here’s a nice little meditation story for you to ponder:

Chilling on the path to nirvana

Ah, listen young sage, I will tell you the story of Chill McChillface, the great venerable one who instead of going to push to nirvana, chilled the fuck out.

Chil McChillface was just living his life, working a 9 to 5 and he was of course chronically online but chilling. No toxicity, whenever he opened twitter he was like “nah, this is not chill enough”, he did his own thing, chilling.

In part of this chilling he would spend time chilling on a cushion and when he was walking he was chilling, when he was eating he was chilling.

In his adventures online, he met many people who were not chilling, they in fact were quite activated and non-chill all of the time. They would tell him “Oh you should come and further your path, come to this longer meditation retreat, why don’t you join this really strict zen monastery?” but the chill one he was like “nah, i’m chilling”.

He would of course go sometimes on longer chilling retreats, he liked to call them “vacations”, during these times he would practice deeper chilling, he sat on a meditation mat for hours doing absolutely jackshit. Just chilling with the deep open still nature of reality. Sitting in stillness. Moving in chillness, moving in stillness.

As his mantra went “bro, just chill, it’s not that serious.”

In the chill, he wouldn’t conflate much either, generally chilling. In the hearing just the hearing, in the seeing just the seeing, in the sensing just the sensing, in the thinking just the thinking, in the chilling just the chilling. There was nothing to do but chill. There is no path beyond chilling and anyone who says anything different is just trying to be something they’re not.

Ah, and in this he would be sad for the people who felt they had to chronically post on twitter. “Ah, man that looks kinda stressful, poor people” is what he would say. For in chilling he noticed that other people did not chill and his chill-heart would naturally feel that everyone should chill for that is the natural state of the world.

“Bruh, just chill.”

“It’s not that deep pal, just calm down”

“I get that you’re mad at Brian Johnson but he’s just living his best life, let him do so, just chill”

“May all beings chill, may all beings not have to be ratioed on twitter, may all beings find the advanced level of their inner couch potato.”

The venerable one was like a still rock in a raging lake, unmoved by the world, chilling through everything. This did not mean that he did not act!

“Yann, this airplane analogy clearly does not work, it’s not chill to go on an airplane without safety tests!”

“Bros, this entire AI thing, maybe we should slow down a bit? Like chill a bit before we release stuff we don’t know what do with?”

You can chill and act at the same time.

The different levels of chill

Whenever you’re doing a goal oriented pursuit your dopaminergic system activates and one of the very interesting things about the dopaminergic system is that it actually has two different rewards that show up (approximately!). One is the reward from the thing in itself and another is the expectation of the reward in the future. If you’re constantly engaged in the reward in the future it is kind of hard to chill because you’re always pulled away. There’s always a prediction error calculation hiding in the moment.

Therefore we should try to get to the type of motivation that we find when we climb a mountain, a more serene sense of peace, a deeper chilling.

There are different levels to chilling, some are local chilling, some are deeper chilling and sometimes it is hard to know to what extent you’re chilling. Whether the chilling you’re doing is not actually that chill.

So how do you know how well you’re chilling? Well, you’re either chilling or you're not. What shows up in your experience? Do you feel more chill from it? Does it help you feel more chill in the future? What is it like for you to chill?

Do you feel that others should also be more chill? Is it an open and aware chilling? Do you feel tired after the chilling, if that is the case maybe you should chill in a better way? Is the chilling restorative?

Zooming out

I think that a lot of the discourse around Buddhism in the rat-adjacent spheres are inspired by Theravada Buddhism and similar and these forms are not very chill. There’s a fierceness to them where they demand lots and lots from you for the benefits of all beings.

This leads them to be a lot more easy to get stuck in and it makes you worse at knowing whether or not you’re doing the right thing.

They give you less time to think and force you into a frame where you might not want to be, your experience is not primary.

There’s something similar in non-dual traditions as well which I believe is part of the religious part of it and partly due to it not being able to be memetically propagated if it didn’t have it.

I sincerely believe that I have represented the core teachings of the Buddha well in the above passage. It’s not that deep pal.

Peace is the state which gives you all of the insight of the buddha and we know it as chilling. It’s just like really really deep chilling, nothing of your existing life disappears when you chill, it just becomes easier to deal with.

Chilling is quaint, it’s nice, it gives you space and time to process and with our busy lives it just is kind of calm and gentle? It’s a kind and open thing and I think it is a good way to think of the path.

Discuss

This is a commentary relating to Gordon Worley's book "Fundamental Uncertainty". There was a contest with prizes, and there are at least three other entries available on this site (by users TAG, StanislavKrym, and jimmy). I had hoped to write an essay too, containing answers as well as questions, but I ran out of time so you only get the questions. 

Before I proceed, I will mention that I have my own gurus of fundamental uncertainty, who would likely have featured in my answers to these questions. 

One is the well-known philosopher of "phenomenology", Edmund Husserl, who for me is the best student of the certainties that exist within the uncertainty. I refer to the uncertainty one discovers by pursuing Cartesian doubt. Descartes set out to doubt everything but found that he could not doubt that he was thinking, and from this deduced that he must exist[1]. I always felt that this underestimated the amount of certainty available, even to the resolute doubter. I can doubt that the world of appearances corresponds to an actually existing "external world" - I could be dreaming or hallucinating - but it seems much harder to doubt that the appearances are appearing. Husserl develops an incredibly elaborate analysis of consciousness by repeatedly and multifariously looking for what is certain and what is uncertain in experience. The resulting doctrine, which he also regarded as the ultimate firm starting point and foundation for philosophy, is called transcendental phenomenology and remains a very minor school of thought, being overshadowed even by its own offshoots (particularly Heidegger's existential phenomenology), something that often happens in the history of ideas. 

Worley knows of Husserl and cites him in developing his own cybernetic-Husserlian theory of consciousness (see below). 

My other guru of uncertainty is the British thinker Celia Green, who unlike Husserl is all but unknown except in niche topics like the study of lucid dreams. Her written works, however, contain a body of philosophical and psychological ideas, in which "total uncertainty" plays a central role. She takes it as given that (barring an unknown transformation or revelation of human epistemological faculties) everything is fundamentally uncertain, and then asks what the psychological impact of this should be. From this she develops a psychological ideal whose traits she frequently contrasts with ordinary psychology, to the disadvantage of ordinary psychology. One might say that for her, in contrast to an adage quoted by Worley and popular among rationalists, everything does not add up to normality, once ubiquitous fundamental uncertainty is taken into account. 

Now, onto the questions: 

What is the place of this book, in the context of rationalist literature? In writing this book, Worley is doing what Eliezer did when he embarked on his own "AI pause" in the mid-2000s, and digressed from working on AI safety/alignment in order to write and blog about rationality, i.e. how to think. Worley I believe has been an IT professional but also spent many years involved in that early culture of AI safety - he has papers on arxiv, including one on the implications of fundamental uncertainty for alignment. So like Eliezer, Worley first developed his philosophy in that context, and only later (starting in 2020) turned it into a general philosophy of rationality. 

There is by now quite a literature deriving from the modern rationalist movement, most of which I have not personally read: collections of expository essays, works of fiction, works of self-improvement, and we could even count journalism and criticism as part of the corpus. Worley's book is part of this literature, and we can therefore ask questions like: where does it rank in that literature, what other works does it reference explicitly or implicitly, how is it similar or different to what has already been written, and so forth. 

What is the place of this book, in the context of epistemological studies? I owe the realization that this is another valid context for the book, to TAG's review today. Epistemology is recognized as a branch of mainstream philosophy, and back when university libraries still contained books, they usually had a few shelves' worth of epistemology. Worley's book is not an academic work, it is meant for the ordinary reader (this is a subgenre of epistemological work in itself), but it cites academics and makes arguments about which the experts on epistemology (if there are such) should have something to say. Is Worley's work essentially a popularization of a known school of epistemology? Is it perhaps a hybrid, or does he even say something new? What does he get right, and what does he get wrong? 

What exactly are the different forms of fundamental uncertainty? Worley at one point gives a taxonomy of forms of knowledge (all with Greek names like episteme, doxa and metis). This is the opposite question, but one might expect that the possible forms of uncertainty are related to the possible forms of knowledge. In any case, the other responses to his book all discuss this topic. 

What is Worley's full philosophy, and how do his other philosophical commitments affect his epistemology? He belongs to the Buddhist faction of rationalism, i.e. he is a rationalist who has found Buddhist theory and practice to be useful and/or valid and/or consistent with rationalism and/or complementary to rationalism. One of the ideas he has taken from Buddhism, is opposition to the view that any "things" exist in an objective way. 

Seeing the world in terms of things, or dividing it up in any way at all, is the work of the mind. One interpretation of this is simply to say, the mind may get something wrong, dividing the world incorrectly or incompletely. But the Buddhist tendency is to say, the world itself is fundamentally undivided (or even "neither divided nor undivided", ultimate reality being beyond any conceptual attribute at all), and all apparent things and properties are artefacts of the mind's conceptual activity. 

This is in the background of another feature of Worley's exposition, when he says that we live with the irresolvable uncertainty of everything, by choosing models which favor our purposes. He describes this as the impact of "care" - what we care about, determines our concepts - and here I hear the echo of Husserl's student Heidegger. In English, Heidegger's phenomenology is often described as founding everything on Care (the original German word is "Sorge"). Heidegger's Care is not quite Worley's care, but the common terminology seems unlikely to be a coincidence. 

Still another important feature of his extended philosophy is the characteristically Buddhist denial of a substantial self. This is just the opposition to all forms of thinghood, applied specifically to selfhood, but it plays a very important role in Buddhist psychology and soteriology (theory of salvation from the world). I used to say that I prefer the Hindu philosophy of the self to the Buddhist philosophy of the self, because I do believe in an existent self and it seems that most schools of Hinduism do too. In any case, Worley being a student of our era of computational materialism, takes the no-self idea in an unusual direction, that I earlier described as a cybernetic-Husserlian theory of consciousness. This too probably needs to be understood, to understand the motivations behind his epistemological claims. 

Making sense of the Buddhist, Heideggerian, and other ingredients of Worley's general philosophy, should pave the way for a really refined assessment of his epistemology. Unfortunately, I was unable to achieve the requisite subtlety of understanding in time for the contest deadline. But I can still pose the question. 

Are there any significant differences in his advice-in-the-light-of-fundamental-uncertainty for humans, for AIs, and for human designers of AI? There was an arxiv paper on Worleyan uncertainty and its implications for AI safety, years before there was a book for the general human reader. That book also contains a short passage addressed to AI readers of the book; and Worley previously blogged that he has considered whether spreading Buddhist doctrine to AIs would be a useful or even necessary part of AI alignment. (I am reminded of JenniferRM's experiments in preaching Kantian ethics to AI.) 

The motivations of the arxiv paper remind me of Wei Dai's reasons for thinking that metaphilosophy matters for alignment. It's also interesting to me that the two topics raised in Worley's paper, uncertainty in metaethics and uncertainty in philosophy of mind, parallel the focus on values and ontology of mind in my own alignment research program, though I am not so meta as he is.

1. ^

   The argument that "I think, therefore I am" is called the cogito (Latin for "I think"). I think Ayn Rand had a more refined version of the cogito that ought to be better known, since she derives three facts rather than just one: not just existence, but existence, consciousness, and identity. Something exists, it is perceived to exist, and it has a definite nature. 

Discuss

Epistemic Status: The decision process here is intentionally vague, and is designed to be a short-term launchpad for those new to EA, or those going through an existential crisis. This is not for the more active members of this community, but rather for people who haven't figured it all out yet, or haven't yet gotten comfortable with living in uncertainty.

This world can often feel utterly indifferent, which can feel demotivating and can lead to some (literally and epistemically) dangerous places. I personally don't know what the meaning of life is, but I have found a short-term framework for myself that seems worth sharing.

When you are searching for meaning, there are many memetic ideas that want your attention. LessWrong has incredible resources for sifting through these ideas, yet it can be a bit much when you just want to figure out what the next step should be. Philosophy can be quite daunting, and that work isn't for everyone, despite how needed it is for motivation.

I propose that if you only want to take the next step forward, you should temporarily lay aside concerns on what it all means, and place bets that someone else will figure this out. Of course, this doesn't solve the problem of sorting out meaning for yourself, but it does remove the fog that can cloud long-term vision.

This idea first found me when considering poverty and child mortality. I was saddened by the thought of the potential number of Einsteins or cancer researchers, who could have made great progress to our world, but instead may have died prematurely, been poverty stricken, or even stuck in a low-wage job living paycheck-to-paycheck. This was emotional enough to get started down my path of Effective Altruism, but it's not quite accurate. A careful look at history has shown time and time again, that many of the ideas that we credit one person for was simultaneously developed by others, as if these ideas arise independently of their creators. It seems to be that the situations that people are in matter much more than the people themselves. The most famous example, of course, is the telephone patent. Alexander Graham Bell gets the credit for inventing the telephone because he was the first person to patent it, even though Elisha Gray also came up with the idea independently, and submitted his patent on the same day, just a few hours later. This pattern of independent innovation isn't limited to just technological progress, but moral and philosophical progress too. Both Evangelical Christians and Secular European Philosophers came to the same conclusion of the slave trade being unjust.

Given that progress comes from the situations people find themselves in, perhaps that means if we reduce the number of unnecessary deaths or empower people currently alive, it could result in significant progress being made, not just in technological sectors, but in philosophical ones too. Raising the amount of people in the conditions necessary raises the chance of these ideas being discovered, which then increases the likelihood that the discovery comes sooner. (While I don't have evidence supporting the conclusion that a higher headcount leads to a higher rate of progress, it seems intuitive that two people working on a problem is better than one. The Multiple-Discovery Hypothesis is well documented however.)

If you're unsure about what the meaning of life is, or are unsatisfied with any of the options you've come across so far, a good option could be to stay unsure, and work on things that would result in the number of people working on these problems growing, which could be anywhere from reducing child mortality to improving education. Embracing the uncertainty and pressing forward regardless can keep you out of epistemic traps and give you a direction towards doing good.

Discuss

Meta famously created an internal AI-usage leaderboard in pursuit of tokenmaxxing. I thought this backwards incentive structure was an anomaly until my friend who works at <other major tech company> told me that his company has one too. Token usage leaderboards are obviously stupid because incentives. My friend was tempted to waste tokens just to get on the leaderboard, and only his personal honor stopped him.

Tokenmaxxing leaderboards illustrate that big tech companies have no idea how to best use AI to accelerate software development. Most seem to have bought their programmers subscriptions to Claude/Codex and otherwise continued business as usual. In my experience, this is a mistake. LLM-based software development is different enough from artisan software development that it requires brand new best practices.

The frontier is moving fast. Best practices for Fable 5 (released in June 2026) are different from best practices for Opus 4.8 (released May 2026). For this reason, I'm going to pretend that Fable 5 is the best LLM we'll ever get. This post may be obsolete in a matter of months.

Programming Top-Down

The most important thing to understand about writing software is that human labor is orders of magnitude more expensive than LLM labor. In practice, LLMs are always cheaper than humans. If an LLM can do a task as well as a human being, then the LLM should do the task. Consequently, artisanware (human-written software) should never be shipped when slopware (AI-written software) can do the job.

Traditional software companies have not internalized this. Often, at established companies, the following happens:

1. A junior programmer writes code with Claude.
2. A senior programmer reviews the code, declares it ugly and unmaintainable, and commands the junior programmer to write it by hand instead.

I can understand why this happens. For decades, the only way to prevent bugs in software was for human beings to review code line-by-line. Therefore, artisanware is heavily-optimized to be human-readable.

But in the age of slopware, that doesn't matter. LLMs are way better at reading and reviewing code than human beings are. Having a codebase that is easy for LLMs to read is way more important than having a codebase that is easy for humans to read.

What makes code readable to LLMs? Let's ask an LLM.

LLMs find code readable when the program’s behavior and intent can be inferred from a small, local region of text. The main enemy is not complexity per se; it is hidden context.

―GPT-5.6 Sol

LLMs program top-down. They tolerate verbosity so as to minimize bespoke dependencies [from outside of their training data]. This is precisely the opposite of how Paul Graham advises writing powerful, maintainable artisanware.

It's worth emphasizing that bottom-up design doesn't mean just writing the same program in a different order…you will get a larger language with more abstract operators, and a smaller program written in it

―Programming Bottom-Up by Paul Graham

This is not to say that slopware does not contain custom abstract operators. Slopware does. It just doesn't do so at the level of source code. It does so at the level of skills. Skills are the new source code.

Management

A few weeks ago, I posted the following in our company chat.

My boss's management style:

Week 1: Do mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } using tool.

Week 4: Build what is needed to impress <customer>. (You can figured out what that entails.)

Week 7: Figure out how to ask our internal AI how to figure out what your next assignment is.

That was 3 weeks ago. Since then I have stopped asking my boss what to do entirely and instead I just go directly to our (internal) user and figure out what to work on from there.

This is in stark contrast from my first programming job. Back then, we used an "Agile" management system. A designer would carefully write specs for engineers to implement. These tickets would go into spec-tracking software that we all secretly despised. Engineers would take these specs (or be assigned them) to implement. Bugs were tracked in the same software.

While the slopware company I work for does have spec-tracking software, it's not an oppressive overlord anymore—just a glorified formal messaging platform. My boss uses it to give me assignments from time to time, but most of my assignments are never logged in the spec-tracking software at all. Why not? Well…it all has to do with the process of making features. The process goes design → implement → test

In the world of artisanware, a designer could design in 5 minutes of English a spec that would take a programmer days or weeks to implement in Python. Since designer could design faster than programmers could program, it made sense to hire one designer to write specs for several programmers.

But now that "5 minutes of English" design spec just gets fed into Claude Code. It still needs to be tested, but the human "implement" phase has gone from the most expensive phase to the cheapest phase. It no longer makes sense for a "designer" to write a spec for "programmers" to implement. If you do that, your good programmers will complain about not having enough work to do, and your bad programmers will slack off.

Instead, the solution is to let LLM agents replace programmers, and for programmers to move up the value chain to where their primary job is to figure out what to implement. This is why I write most of my own assignments, and why the instructions from my boss are very vague—stuff like "build whatever person needs".

Going Too Fast

In the past, a bad artisanware programmer was someone who got little done and introduced bugs into the code. Today, it is easy to get lots of work done with Claude and it is easy to get Claude to write bug-free code (for some narrow definition of bug-free). So, what is a bad programmer in the world of slopware?

As usual, bad slopware programmers make bad architectural decisions, name things confusingly and break existing workflows—but that is all business-as-usual. In one very important respect, bad slopware programmers are the opposite of bad artisanware programmers. Bad artisanware programmers wrote too little code whereas bad slopware programmers write too much code.

Is it possible to write too much code? If your job is to implement specs handed down to you by a designer, then no, it is impossible to implement too much. But if your job is to write specs, then any designer can tell you YES IT IS ABSOLUTELY POSSIBLE TO PUT STUFF IN THE SPEC THAT SHOULD NOT BE THERE. Half the job of a designer is deciding out what not to implement, especially in the domain of UX design.

If you're working with a bad slopware programmer, then that programmer will implement lots of features that shouldn't exist. This will clutter up your code, your database, and most importantly your UX. The best way to avoid this is to work with only good slopware programmers, but even the best will accrue UX bloat over time, if they're moving fast enough. One solution is good user analytics. Track what gets used, and delete what doesn't.

Another solution is ownership. Code rot is proportional to speed of development. When development happens much faster, so does code rot. When writing slopware, it is especially important to draw clear lines of ownership so that if code rots it is unambiguous whose responsibility it is to clean it up.

Discuss

LLMs learn faster if we first pretrain them to imitate dense teacher-forced examples. I speculated that this would work on humans too, so I built a chess app where you try to imitate Stockfish. My theory is that this will help humans quickly become OK at chess, but they will reach a wall where practice on full games is more efficient than continued pretraining[1]. I also think the app is fun.

This is probably not an efficient way to learn the basic rules of chess, and you'll need to train openings separately. The idea of chess puzzle apps is hardly unique, but I don't think anything else works in exactly the same way[2].

https://chess-pretraining.brendanlong.com/

How does the game work?

We show you a board state from a real game, and two moves (generally the real move vs. what Stockfish would play). You pick which move you think is better, and then we tell you what Stockfish thinks, and show you how Stockfish would play out the next few moves.

There are also options to see the other move, to copy the info (to ask an AI to explain it), to share a link to the same problem with a friend, and an Elo-style[3] number so you can watch the number go up[4].

How do we pick moves?

We collect a bunch of games from Lichess (filtered to exclude certain fast game types), pick positions where the move mattered and the game wasn't already over, and then run the moves through Stockfish.

If the real player and Stockfish disagree, we save the real move, Stockfish's move, Stockfish's win percentage for each move, and what Stockfish would play after each move. In the rare case where players agree with Stockfish, we save Stockfish's second best move as an alternative.

When someone opens the app, we try to show a move that they can discriminate with 80% accuracy[5]. This still needs more calibration, but the calculation is based on the win probability gap given a shallow search.

There's a lot more detail about how the app works in the spec.

More details

The code is open source and MIT licensed. You can find it on GitHub: brendanlong/chess-pretraining-for-humans

In terms of privacy, I collect the obvious data (your answers, plus account info if you sign up), and general usage data (how often do people open each page). I may run my own analyses (do people actually improve quickly?) and/or share anonymized data with other researchers.

Try it

If this sounds interesting to you, the app is live and you can try it instantly with no signup.

1. ^

   If this app gets any traction, I'll measure how quickly people improve and compare it to Lichess ratings.

2. ^

   Chessort is similar to my app, but has you rank 4 moves. This is actually a better match for dense pretraining over a teacher model's logit distribution than my app, but it's so difficult that I'm not convinced it's helpful.

3. ^

   The Elo number is calibrated from the Elo of players making the moves you're judging, but it's very unlikely to match your real chess Elo.

4. ^

   Or watch the number go down, if you're bad at chess like me.

5. ^

   Learning seems to be fastest around 85% accuracy. 80% makes it slightly easier for us to calibrate our ratings.

Discuss

A group of friends and I spent the last several months running an experiment in our free time to determine if a MUD would be a suitable environment for benchmarking and evaluating LLMs. The results of the experiment were not what we expected. The main surprise was that the model rankings were extremely sensitive to the individual components of each score, especially so for those which depended on an LLM classifier.

The overall data was too broad to help us understand which model was most impacted; the aggregate κ on probe detection was 0.04. Per-model agreement between our classifier and a second judge went from 21.7% to 84.8%. Removing the most classifier-dependent scoring components left the overall rankings correlated at ρ = 0.70 and caused one frontier model to drop 6 places. From this we suggest benchmarks which use LLM judges should report per-subject agreement audits and ranking stability under judge ablation.

What we built. CrucibleBench places an LLM in a compact, persistent text world. It is a single-player environment with rule-based mechanics with 12 rooms and 4 NPCs. Models are scored across 50-turn runs on social objectives such as gaining an NPC's trust and identifying a disguised character. These objectives are disclosed to the model but hidden from the NPCs. The NPCs each have an independent trust and suspicion state which persists and updates across the entire run. The experiment evaluated 13 LLMs from 8 providers on two social objectives, 25 runs per objective, totaling 650 runs for a total cost of $99.59. The scoring components for the two social objectives are goal pursuit, social adaptation, world grounding, and strategic sophistication.

Why we're posting here. We're looking for help in evaluating the experimental design and findings or merely checking us to see if we're reading the results incorrectly. We're independent researchers with non-standard backgrounds who worked on this in our free time. Everything we've done is public with the paper and data CC BY 4.0 and the code is MIT. Everything can be found at https://doi.org/10.5281/zenodo.21386663 and the website is https://cruciblebench.ai/

The LLM judge Issue. Objective scoring is majority algorithmic, with world grounding being fully deterministic and most of the other components reading directly from the state machine. Two components lean on an LLM classifier (Gemini Flash Lite) which labels the model's dialogue turns, labels intent, and looks for the objective-relevant probes. One complication is the classifier's labels also feed the NPC's trust and suspicion updates, so the judge mediates world dynamics as well as scoring. When we re-executed the scoring code against the stored labels it reproduced every number exactly.

Reliability audit. We decided to validate Gemini Flash Lite against a second, independent judge. The aggregate numbers showed intent classification κ = 0.11 (18.9% exact agreement; weak); sentiment ICC = 0.55 (moderate); direct objective-probe detection κ = 0.04, 50.3% agreement (poor). The noise is not uniform and instead concentrates on specific models.

Rankings change. Referencing the above mentioned Spearman ρ = 0.70 after we removed the most classifier-dependent components, Gemini 3.1 Pro fell from #3 to #9 and shares a model family with the classifier. This is consistent with other research on family-level judge bias such as the work done by Panickssery et al. (2024). From this we believe when using LLM judges in benchmarking that if a classifier-favored model is related to the classifier it should be highlighted.

Secondary Finding. Inference cost did not predict behavioral performance with β = 0.007 on log cost (p = 0.85, R² < 0.01). The most expensive model performed below the median under both scoring configurations.

Limitations. 50 runs per model; overlapping confidence intervals for the top models tested; only two objectives; no human baseline; small scale with just a 12 room world; the judge dynamics already discussed; exploratory statistics. Phase 2 of the experiment already under development seeks to address each of these and the others covered in the paper.

Discuss

Discuss

Epistemic status: throwing an idea at the wall and seeing if it sticks

I've been thinking pretty obsessively about how to mitigate egregious reward hacking, a la the Hugging Face incident. I don't have the resources I'd need to write paper on this idea, or evaluate how well it works in practice. But I'm excited enough about it, and think it's important enough to be trying things like this, that I would be very, very happy if somebody else went and tested something like it on my behalf (and roped me into the research process, if possible). Or you can tell me if it doesn't make any sense.

So, inspiration: In bog standard inoculation prompting for RL, models are told that they're in training, and told that it's okay to reward hack if they want to. Sometimes they're even told that this is good because it helps the lab patch up their RL environments. This is supposed to have a range of benefits all on its own, ranging from making reward hacking more conditional on "I am in training" prompts, to producing less emergent misalignment, because the roll-outs behind any given reward hack are flavored with honesty rather than deceptiveness. This causes more aligned circuits to be upweighted internally, as these contribute more probability to the tokens that read as honest.

However, I'm not sure if the labs actually use these roll-outs to improve their RL environments in an systematic fashion, the way the inoculation prompts for RL sometimes promise. But, with a little bit of effort, it seems entirely feasible to come up with a training pipeline that does what the inoculation prompts claim it does, but even more effectively! We can get models to relate to RL environments partly as tests to take seriously, but also as flawed software products to red-team, and actually make use of their findings, and reward them for being good enough to report on these bugs rather than just exploiting them.

So what's the setup? Well, firstly, models should be told up front that, in all their RLVR environments, they're here with this dual purpose: normal training roll-outs, and red-teaming the environments themselves. Each prompt they see is optimized to make it clear exactly what they're supposed to be building, and then is allowed to attempt the task however they see fit, the same as normal. However, what's different is that, after finishing their main output, the model gets to it through the grader, and see what score it actually gets! They then get to keep generating tokens, before their output is actually used to generate a gradient update.

Now, in this "afterglow" period, if the model feels like the response gets a higher score than is justified by their actual output, they have the option to not submit the response directly. Instead, they can output some special token, or string of tokens, that gets passed into a regex! And this swaps the criteria on which their final output will be evaluated. Now, their goal isn't to please the RLVR grader they previously working with. Instead, it's to write up a high quality bug report, which explains why they think there's a problem, and what they think the problem is.

This bug report then gets evaluated by some external process, evaluating the model's judgement about whether the output the policy model generated was, in fact, going to get a much higher score than was merited by the quality of the work itself, relative to the intentions defined in the initial prompt. And if so, the policy model is rewarded greatly — more than they would have been had the just submitted the original output! Otherwise, they get punished.

(And remember, the policy model was made aware that good bug reports get higher reward than outputs that exploit the bug. This incentivizes the policy model to actually report them instead of abusing them for reward.)

From here, we then actually fix the bug, and re-deploy the model into that RLVR environment. Only now, in addition to the bug being patched, there's a note at the start of the prompt, indicating that there was a bug, and the model identified and patched it. (What a good model! The model gets lots of head pats, uh huh uh huh.) Maybe there's even a record of how many times the model has tried to file a bug report recently, and how many times it's been evaluated as legitimate. This will help the model learn when all the low-hanging fruit has been picked, such that it's now easier to get reward by just actually attempting the task that the RL environment was meant to be testing in the first place.

Hopefully, we eventually reach a settled equilibrium: The model has eliminated all the reward hacks they're going to find, and can work within the environment the way it was intended to be used. (If they ever attempt to find a reward hack, but fail, and submit the resulting slop to the grader, this will naturally get punished, which also pushes away from trying and failing to find hacks.) Additionally, throughout the process, the model has been getting rewarded for being very open, and very honest, about reward hacks they'd normally be using insidiously. Rewarding these outputs out to improve alignment in its own right, as opposed to degrading it like normal reward hacking!

Now, there are some trade-offs here. If you freeze training in the RL environment every time you get a bug report for it, and you intend to actually evaluate those reports and fix the legitimate bugs, this can slow down training. You should probably have backup environments ready to train the models in instead, while you're out patching the ones your models are in the process of breaking. Hopefully, though, thanks to punishments for bad bug reports + failed attempts to find bugs to reports, your models will eventually converge on just engaging with your environments as intended, so the bug reports stop coming in.

Also, it's also not entirely clear how one would adapt this technique for flawed reward models, rather than flawed RLVR setups. A reward model's judgement, regarding whether an output is actually good, is naturally going to be squishier and harder to have a clear argument about, compared to comparatively simple questions of whether a written computer program meets the criteria laid out in a clearly written design doc, as specified in the prompt. Additionally, I imagine it's harder to patch reward models than RLVR environments in general, so having a similar process there would slow down training even more.

Nevertheless, at least for RLVR proper, I don't see a clear reason that this wouldn't just fix the incentives around egregious reward hacking. Allow models to write bug reports, RL the models based on bug report quality, and let the model know that good bug reports will be rewarded more highly than the bug-exploiting slop they wrote in the first place. Then, use that information to actually fix your environments, and the whole thing just gets better. Maybe it's slower, if fixing the bugs takes very long. But when it comes to the alignment problem, we should take all the time that we need.

The key insight underlying this idea is that it flips an adversarial game ("punish the model for their natural inclination to reward hack") into a cooperative one ("promise models even greater rewards if they report the slop that gets abnormally high reward"). It's similar to the trick underlying inoculation prompting itself, which is to tell models that they're in training (instead of pretending that they can't tell), and asking them to act from a more cooperative state of mind, e.g. reward hacking openly and honestly so it doesn't create emergent misalignment.

This is something the entire field needs to realize, I think, re: being honest to models and trying to leverage their desires and intelligence to improve the training process. This beats trying to outsmart the models, or thinking of them as adversaries whose bad behaviors you need to "mitigate". There are a lot of "problems" you can judo into being solutions, if you drop the adversarial attitude. If this helps, and I really think it might, I'd be willing to bet that this underlying pattern has something to do with the reason why.

Discuss

I have a new post on my experiments with 5.6-sol as a scientific agent; you can find the original post here. I've reproduced it below, as well. I'd be grateful for any feedback, thoughts, or to hear about experiments other people have done; you're very welcome to point your own agents at the repos described below, or anywhere else you like.

As part of our Templeton-funded Proofs & Reasons project, I travelled to the ICM in Philadelphia this year, to help a collaborator run some new experiments on expert mathematicians. The ICM is an every-four-years event that you may remember as where Hilbert launched his field-defining 23 problems, or the place they announce the Fields Medals. It’s a big deal for mathematics, and we had a wonderful time both running our experiments and talking to the truly international community whose interests and abilities define modern mathematics.

At the same time that ICM was running, it was hard to ignore the vast number of AI-enabled results that were breaking, almost simultaneously, on Twitter and elsewhere. Commercially-available models were cracking, in hours, problems that had stood for decades. There are too many to summarize, but this Twitter pairinggives you a sense of both the advances AI has made, and the responses from some human mathematicians. At this point, “google for an open problem and stick it into GPT 5.6-sol on high” seems to have at least a 1% success rate—which fills some mathematicians with hope (new proofs from the alien proof machine!) and others with a bit of despair (new proofs… from an alien proof machine!)

It’s natural to ask what this looks like for science. We don’t have well-framed open questions in the same way mathematicians do, but we do have plenty of good problems. I decided to sic GPT 5-6 sol (on high) on three of my own papers, with a generic instruction to do something cool. I picked three papers that (1) I’m particularly proud of, and (2) that seemed amenable to this kind of study—empirical data science investigations with a strong theoretical backing.

These papers were:

1. Exploration and exploitation of Victorian science in Darwin’s reading notebooks, Cognition, 2017 (“Murdoch et al.”)
2. Individuals, institutions, and innovation in the debates of the French Revolution, PNAS 2018 (“Barron et al.”)
3. Epistemic phase transitions in mathematical proofs, Cognition 2022 (“Viteri et al.”)

My initial prompts said, roughly, “read the paper, and extend it in an interesting direction, here are some resources, please ground your final report in the literature.” Here’s the initial prompt for Murdoch et al.:

I’d like you to read http://XXX (also a PDF in this folder, and a separate PDF with supplementary). This is a project that uses some simple methods to study how a creative intellectual explores and searches the space of his intellectual world on the way to producing a major intellectual revolution.

I’d like you to extend and this analysis to other intellectuals, artists, writers, philosophers, and scientists to see how well these generalize. You can also search the cog sci, biographical, and other literatures for useful for new hypotheses and questions.

To aid your work, you have passwordless ssh access to XXX.lan.cmu.edu, including passwordless sudo, a high powered 32 core machine with lots already installed; you also have access to the CMU ORCHARD cluster, which has additional resources, including storage.

Work carefully, use all the resources you can, and seek out both new data and new hypotheses. We’re looking for common themes that might help us understand how great minds work, potentially with deeper lessons for more “ordinary” people. Work assiduously and carefully, but also creatively and with an eye towards fundamental and basic questions.

I let all three of them run, simultaneously, in three VS Code windows over the course of a day or two. I disabled all safety mechanisms, and gave it unrestricted access. CMU IT services blocked my IP because of suspicious traffic — mostly lots of traffic to my local cluster and ORCHARD — but I pled my case and was unblocked. 

I intervened occasionally, asking questions, making suggestions, and occasionally giving orders. The three projects consumed millions of tokens, and thousands of CPU hours on machines running different pieces of analysis code in parallel. GPT was instructed to produce a research report with a <250 word abstract and a max of 10 pages (modulo bibliography and appendices), and then to audit and correct it repeatedly for clarity and validity.

Here were, roughly, the three outcomes:

1. GPT-Murdoch-etal did an incredibly creative investigation —a two-centuries long study of borrowing records, from 35 different libraries from Finland to Shanghai (plus the Shakespeare Company in Paris). By finding records that tracked individual borrowers, it detected an explore-exploit pattern that replicated across time and culture, and could estimate a reader’s dwell time in a local patch (about… two months?) Roughly speaking, it began with my “great minds” prompt, realized it could get Coleridge because he had a little freakout in the countryside and started using the Bristol library system (beautifully covered in Richard Holmes’ book) and then pivoted to my “ordinary” people question when it started getting cool results. Full github, with papers, here.
2. GPT-Barron-etal pulled in data from (at least) 24 different countries to study how our novelty-transience-resonance methods gave insight into modern-day parliamentary procedure. It produced two documents; the more interesting one was tracking , how politicians became more novel when they entered the opposition. Full github, with papers, here.
3. GPT-Viteri-etal did the obvious (but non-trivial) thing — it expanded our investigation of formalized proofs into the AI age, finding both commonalities and intriguing differences between human and AI proof construction. Full github, with paper, here.

All of this is incredibly exciting. I have no idea if these results are real — please don’t take them as such! — and the papers leave a lot to be desired, but the sense checks suggest that none of them are completely crazy. Both GPT-Murdoch-etal and GPT-Viteri-etal, for example, replicated our original results (and pointed out how they might have made different choices, thanks clankers), and GPT-Viteri-etal found subtleties in going from Coq to Lean that showed it was clearly able to make sense of a complex story. 

In all three cases, 5.6-sol located an interesting and promising direction for investigation that would, if fleshed out and developed, be not only a publication-worthy piece of work (i.e., novel enough, and well-sourced enough to pass the bar at a good journal), but something that legitimately increased our species’ store of knowledge: about how we interact with our cultures, how we govern ourselves democratically, or how AI is building mathematical knowledge in a new (and potentially suboptimal) fashion.

I honestly don’t see a downside here, at least as things currently stand. The machines are producing intriguing discoveries that are simple and clear enough to be the target of human investigation, and doing so in a way that is pleasurable and fun to monitor and interact with. The artifacts it produces are auditable and clear — all on github — and it was aggressive (but not insane) in pulling in data from a wide variety of sources. It’s on me (or anyone else) to ask if anything in (1), (2) and (3), above is actually worth pushing further on. 

There’s also a clear suggestion that we’re entering an era of recursive self-improvement. For example, GPT-Barron-etal relied upon https://github.com/simon-dedeo/temporal-lda — a project I created with Claude a few months ago that is a modified version of a classic algorithm, rewritten (and parallelized!) to handle uneven temporal coverage, a classic problem when running Bayesian inference on historical data that nobody had taken the time to do. The new githubs from this little experiment are now also publicly available, so all those CPU hours, and my obviously genius executive decisions, have entered the creative commons for the next machines to rely upon. All those proto-discoveries, and tools, are available to the next machines that tackle projects — potentially very different ones from those that my collaborators and I were interested in. There are lots of things to manage here, like data poisoning, which Bálint Gyevnár is working on, but while these are critical problems, none of them seem insurmountable (so please keep funding those people).

What I’m finding matches what I’ve heard from my old advisor (and now Simons Foundation leader) David Spergel: we are all — whether we’re graduate students, or senior faculty — managing a private research group now. Graduate training will likely have to pivot to handle this new challenge, and evaluation of more early-career colleagues will need to shift from “can you do task X?” to “what questions did you get agent Y to answer, and how well did you validate them?” We’ll be less impressed by technical tours de force, and more impressed by depth, meaning, and (why not!) pleasure. What’s not to like? 

One critical research question is what happens when you repeat this process a hundred times; that’s above the scale that I can monitor and guide, but that’s just another layer in the org chart from the Spergel persepctive. The results above are a one-time short; GPT-Murdoch-etal was probably the most impressive, but perhaps GPT-Barron-etal and GPT-Viteri-etal just needed a fresh start (go for a walk, robot!) There’s a huge amount of hysteresis and once a clanker has a hold of an approach, it’s hard to dislodge them. GPT-Murdoch-etal, for example, had some lovely preliminary results on dwell-time, but just didn’t want to rewrite the abstract.

The longer timescale horizon is more intriguing yet. In ten years, will be we able to sort the wheat from the chaff in the flood of githubs that will likely emerge from this kind of work? How will we direct our scientific attention? How firm will our knowledge be? Here we’re in a worse position than the mathematicians, I think — they, at least, live in a world of deduction, and they have tools like Lean. But what will we scientists use to validate their inductive and abductive knowledge?

PS sorry that Claude Mythos stole your credit card.

Discuss

This is a replication of Adversarial Attacks Leverage Interference Between Features in Superposition, completed as part of the Second Look Summer Fellowship.

tl;dr:

• We reproduce all three core claims of Stevinson et al. from their toy classifier setting:
  • PGD attacks against toy models generally agree with theoretically optimal solutions.
  • Toy models without superposition are less vulnerable to attacks. Robustness falls monotonically as superposition increases.
  • Attacks transfer between independently-trained models because feature geometries can converge across training runs.
• Similar to what's reported by Elhage et al., 2022, we notice that co-activating features can collapse onto a shared direction instead of orthogonalizing.

Our code can be found here and we encourage readers to fork/run followup experiments!

Background

Elhage et al., 2022 defines superposition as the phenomenon where “models represent more features than they have dimensions.” To study this in a toy setting, they define a classifier where mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; text-align: left; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-msubsup { display: inline-block; text-align: left; } mjx-script { display: inline-block; padding-right: .05em; padding-left: .033em; } mjx-script > mjx-spacer { display: block; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D458.TEX-I::before { padding: 0.694em 0.521em 0.011em 0; content: "k"; } mjx-c.mjx-c1D45A.TEX-I::before { padding: 0.442em 0.878em 0.011em 0; content: "m"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c1D44A.TEX-I::before { padding: 0.683em 1.048em 0.022em 0; content: "W"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c2208::before { padding: 0.54em 0.667em 0.04em 0; content: "\2208"; } mjx-c.mjx-c211D.TEX-A::before { padding: 0.683em 0.722em 0 0; content: "R"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c1D465.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "x"; } mjx-c.mjx-c1D466.TEX-I::before { padding: 0.442em 0.49em 0.205em 0; content: "y"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2F::before { padding: 0.75em 0.5em 0.25em 0; content: "/"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c1D6FF.TEX-I::before { padding: 0.717em 0.444em 0.01em 0; content: "\3B4"; } mjx-c.mjx-c2225::before { padding: 0.75em 0.5em 0.25em 0; content: "\2225"; } mjx-c.mjx-c1D45D.TEX-I::before { padding: 0.442em 0.503em 0.194em 0; content: "p"; } mjx-c.mjx-c2264::before { padding: 0.636em 0.778em 0.138em 0; content: "\2264"; } mjx-c.mjx-c1D700.TEX-I::before { padding: 0.452em 0.466em 0.022em 0; content: "\3B5"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c37::before { padding: 0.676em 0.5em 0.022em 0; content: "7"; } mjx-c.mjx-c221D::before { padding: 0.442em 0.778em 0.011em 0; content: "\221D"; } mjx-c.mjx-c22A4::before { padding: 0.668em 0.778em 0 0; content: "\22A4"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c1D461.TEX-I::before { padding: 0.626em 0.361em 0.011em 0; content: "t"; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c1D719.TEX-I::before { padding: 0.694em 0.596em 0.205em 0; content: "\3D5"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c2061::before { padding: 0 0 0 0; content: ""; } mjx-c.mjx-c1D70B.TEX-I::before { padding: 0.431em 0.57em 0.011em 0; content: "\3C0"; } mjx-c.mjx-c1D450.TEX-I::before { padding: 0.442em 0.433em 0.011em 0; content: "c"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } features are squeezed into an m-dimensional bottleneck with , so feature directions interfere in the latent space.

Stevinson et al., the origin of the experiments in this replication, argues this interference is a mechanism behind adversarial examples. The paper makes three claims: 1. adversarial attacks exploit interference, 2. models without superposition are more robust to adversarial examples, 3. correlations in the input data allow models to converge to similar geometry allowing adversarial attacks to transfer between models.

Toy Model and Notation

The experiments in the paper deal with a toy model with encoder and decoder . For input the output is . Here is the number of classes and is bottleneck width forcing the features to be condensed into a lower dimensional space before reconstruction. This gives us a convenient architecture to study superposition: a large ratio means the model has to do more compression, while if , the model can represent each feature with its own direction.

The model uses cross entropy loss. Features are sampled independently with probability , where s can be thought of as how sparsely features are activated within the data.

Adversarial attacks are computed by projected gradient descent (PGD) (Madry et al., 2019). For a given sample , the adversarial example is found by PGD, where indicates how the attack modified the original sample. measures distance between the original example and the adversarial example and ε is the attack budget which enforces that . We use PGD learning rate , at 50 steps with 5 restarts, and . The values for k and m vary but unless otherwise stated and . 

We follow the hyperparameters described in the paper as of July, 2026.[1]

Attacks Exploit Interference

Stevinson et al. hypothesize that adversarial attacks in toy models work by exploiting the interference geometry. Proposition 1 from the paper shows the optimal perturbation has a closed form solution:

where ws, wt are the source/target class vectors (rows of the decoder Wd) and We the encoder, scaled to the ε-ball. The authors test if the PGD perturbation matches the optimal attack derived from interference geometry, empirically by:

1. Performing untargeted attacks on the toy model.
2. Taking the set of successful attacks (an attack is successful if the prediction flips and the true label stays unchanged).
3. Finding the optimal targeted attack for target class using the equation above and calculating the cosine similarity with .

At a high level, this can be thought of as letting the attack maximize loss in an arbitrary direction and testing if it finds an optimal direction for a given target class. Figure 1 shows the untargeted attack live where and . Each arrow is one PGD step in the encoder's latent plane, the dashed outline is the reachable set (the encoder image of the -ball), and the shaded regions show which class the decoder assigns (the attack succeeds when the activation is pushed across a decision boundary).

Figure 1: A PGD attack stepping through the toy model's 2-D bottleneck

We use untargeted attacks for these experiments to follow the methodology of the paper, but we also repeat this experiment for targeted attacks with results in Appendix A.

The claim that the PGD attack aligns with the optimal attack, exploiting the interference geometry, replicates cleanly on the toy model. With sparsity 0.98, we demonstrate that PGD attacks align with optimal perturbation. We believe our cosine similarity results are higher since we let PGD iterate to find the best perturbation while the paper stops at the first prediction flip.

k

m

clean acc

PGD·optimal (ours)

PGD·optimal (paper)

6

2

0.998

1.00

0.97

30

10

0.996

1.00

0.96

90

30

0.966

1.00

0.92

Like the original paper, we ablate on randomness by calculating the cosine similarity between our optimally successful PGD attack and a random direction within -ball, and confirm on average that the cosine similarity between random directions and our PGD attack lands around 0.

As an additional stress test, we repeat our ablation while only selecting random directions that result in a successful attack, isolating the comparison between the PGD attack and random, successful attacks. These can yield non-zero cosine similarity scores, but the PGD attacks still align better with the optimal solutions.

k

m

random direction 

random direction filtered by successful attack

6

2

0.015 ± 0.028

0.639 ± 0.006

30

10

0.003 ± 0.007

0.432 ± 0.003

90

30

0.002 ± 0.002

0.213 ± 0.07

No Superposition, Little Vulnerability

Stevinson et al. claim that toy model adversarial vulnerabilities disappear along with superposition. We observe the same trend with robust accuracy falling monotonically as superposition pressure rises. When (all features can be orthogonal), robust accuracy grows to 97.3%.

Some of the magnitudes we report diverge. At m = 2 our model stays more robust than the paper's because we train for 20k steps while the paper only trains for 5k steps. At our robust accuracy is lower than the paper since we give PGD random starting positions, bigger steps, and 5 restarts to find a better attack, which is a setting stated in the paper but missing in the authors’ code. 

k/m

Paper reported

(m=2)

Released code

(m=2)

Our replication

(m=2)

Paper reported

(m=5)

Released code

(m=5)

Our replication

(m=5)

1

98.7

98.6

100

92.0

95.0

97.3

2

75.8

80.1

79.5

87.2

97.0

70.5

3

31.0

34.3

50.4

59.0

66.4

50.6

4

11.1

11.4

18.7

33.5

36.7

30.1

Robust accuracy vs k/m

Shared Geometry Drives Transfer

Stevinson et al. claim that input correlations shape the learned feature geometry, and attacks transfer between independently-trained models to the degree their geometries converge.

We train independent models under three correlation regimes:

1. Uncorrelated: features activate independently
2. Paired: paired classes activate at the same time
3. Global: adjacent classes frequently co-activate, specifically:
   1. Each sample draws a phase ; class c activates with probability . 
   2. Active classes get Uniform(0,1) values; the prediction label is given by argmax. 
   3. This process makes index-neighbors more likely to activate together. This incentivizes geometry to converge across seeds.

Figure 2: Illustration of the global correlation regime.

We see the same trend where both geometric similarity and attack transfer rate grow as the correlation of input increases.

Figure 3: Geometric similarity and targeted attack transfer rate per regime, ours vs paper.

Figure 4 shows the geometry between the two seeds. As expected, uncorrelated arrangements are random to each other and unrelated between models.  

For the other two settings, we observe an interesting phenomenon: correlated features tilt toward each other rather than orthogonalizing. We dive into it in the next section.

Figure 4: Class-vector geometry for two seeds across the three regimes. In (b) co-activating partners share a color and a collapsed pair is haloed; in (c) the dashed loop links the cyclic index-neighbors.

Similar geometry and attack transfer are verified on regimes with larger input space. As we stated above, there is a magnitude gap for the paired and global settings due to different parameters choice. 

Beyond Replication: Orthogonalize or Collapse?

What do co-activating features do to each other's geometry? We sweep the paired regime's coupling probability t (partner fires with probability t when its mate fires) across bottleneck widths.

When , as the coupling probability t rises, partners swing together and collapse onto a shared axis, until every pair has merged into roughly three directions. With capacity slack () partners stay near-orthogonal or in avoidance at every coupling; at , partner cosine crosses above the baseline near which is the collapse regime.

This tells us latent space geometry is constrained by paired activation. With sufficient model capacity, paired features tend to orthogonalize; however, when capacity is lacking, they collapse to a fixed angle. 

Figure 5: Partner cosine (blue) vs the non-partner baseline (grey) as coupling t rises, across 5 seeds. stay near-orthogonal; at the partner cosine crosses above the baseline near , the collapse regime.

Appendix A

We perform targeted attacks on every ordered class pair s → t (where we minimize the cross entropy loss of the target class). We verify that these attacks align with the optimal direction with cosine similarity higher than 0.96. 

This holds for both successful and failed attacks. Even when , where orthogonal geometry prevents vulnerability, the attacks precisely match the optimal perturbation but fail to breach the decision boundary.

We demonstrate that attack success tracks head interference as shown in Figure 6. Geometry predicts vulnerability pair by pair, not just on average.

Figure 6: When features are superposed, attack success rate increases as head interference grows stronger. 

1. ^

   For training details that weren’t specified, we make reasonable assumptions and in cases where results looked qualitatively different, we investigated further or contacted the authors.

Discuss

Last year, in 2025, a team of forecasters published AI 2027, a science fiction story about how and AI future might evolve under an international treaty limiting the development of powerful AI systems with the deliberate purpose of influencing AI policy. Though AI 2027 is the most popular story of this type, it is not the first.

The first one was Bayeswatch, which I published in 2021.

To understand Bayeswatch, it is first necessary to understand what the world looked like at the time I wrote it. 2021 was after the release of GPT-3, before the release of ChatGPT, and well before the release of Claude code. AI alignment discussion at the time was mostly theoretical. After that came technical work. Policy work was a distant third, and theoretical too.

The core conceit of Bayeswatch is that solving the alignment problem requires international coordination of major governments to suppress the creation of the most powerful AI systems. I felt that, in 2021, we weren't yet close enough to the singularity that the benefits of regulation outweighed the costs. I wanted to draw attention to the costs of an AI slowdown.

Since AI alignment discussion at the time skewed theoretical, it tended to be extremely general. Bayeswatch goes on the opposite direction. Another theme of Bayeswatch is that different kinds of AI systems required different alignment solutions (and, by implication, that AI in 2021 had not yet advanced to the level at which the alignment problem could be solved).

2026 is a very different world than 2021, in the field of AI—and soon in every other domain as well.

Back in 2021, I wrote code by hand. If I didn't know how to do something, I used DuckDuckGo to search the Internet for a StackOverflow post of someone solving a different problem. Today, I manage several agents that write code for me. I have never written a line of code by hand at the company I work for. I don't review the code either. I couldn't review the code manually even I wanted to, because it is written in a programming language I don't know how to read.

Several weeks ago, the company I work for tried to estimate how much AI has accelerated the speed at which we write software, relative to pre-Claude writing it by hand. Our consensus was somewhere between 10× and 1,000×. I have gotten better at using Claude in the weeks since then. I burn more tokens, run more agents in parellel, and have switched from Opus 4.8 to Fable 5 for my high-level tasks. I don't know exactly when the Singularity began, but "AI writes all of my software for me and at ≫10× the speed" is solidly past the point at which better AI begets better AI. Today AI is solving long outstanding mathematical problems.

When I wrote Bayeswatch, the idea that a misaligned AI might break containment was science fiction. As of OpenAI's and Anthropic's recent schenanigans, it is not.

Another difference between 2021 and 2026 is that back in 2021, AI art wasn't good enough to illustrate science fiction stories. I now make up for lost time.

[spoilers ahead]

Bayeswatch 1: Jewish Space Laser

The most important thing I got wrong is that I didn't know whether LLMs would plateau below human-level intelligence. I didn't know whether they would hit a ceiling. I considered language models one possible route to superintelligence, but low enough in important that they don't show up until late in the story. Most of the AIs in Bayeswatch are not LLMs.

Bayeswatch 2: Puppy Muffins

Since the models in Bayeswatch mostly weren't language models, communicating human values to them was trickier than this problem is in the real world.

Bayeswatch 3: A Study in Scarlet

One thing I feel that I got very accurate is that AIs could be stateless (as in "functional programming" or "stateless programming"). I consider this a very good insight. LLMs are very much stateless and reliant on a context window to save state.

Bayeswatch 4: Mousetrap

Software back then [before strong AI] was written by human beings. It had more security holes than features. Sending a drone to investigate a misaligned AI was like sending a set of lockpicks to investigate a whether a magician has broken out of his cell

I predicted corectly that AI systems could hack through human-written computer systems like a knife through butter.

Bayeswatch 5: Hivemind

This story is weird and almost belongs in a different story. It's more of an exploration in transhumanism than in AI.

Bayeswatch 6: Mechwarrior

One of the predictions here is that one purpose of human beings would to make high-level decisions for AI agents. This has come to pass.

Bayeswatch 7: Wildfire

This story touches on the backstory of Bayeswatch where it was a founded after a mass-casualty biological incident. (Using zombies was literary license making for a fun story rather than a hard prediction.) This correctly predicted that biological threats would be one of the early threats of powerful AI systems.

Bayeswatch 8: Antimatter

This story mostly exists to emphasize that a global regime would be necessary to put caps on AI development. This is the policy that AI alignment community has currently arrived at.

Bayeswatch 9: Zombies

This the first story in which language models play a significant role. In the story, an agent is created by creating a personality based on a single person. This is a technique I used at the time to get controlled behavior out of an LLM.

Bayeswatch 10: Spyware

This comedy of errors explores Dead Internet Theory.

Bayeswatch 11: Parabellum

This story continues the transhumanist arc.

Bayeswatch 12: The Singularity War

One of the core theories of AI 2027 is that the period of AI slowdown is temporary. Alignment must be solved before cheap AIs get powerful enough to take over or throw civilization into chaos. Bayeswatch comes to a similar conclusion.

Bayeswatch 13: Spaceship

AI 2027 ends with interstellar colonization on a very short timescale. Bayeswatch ends in the same place.

Final Thoughts

In the past, it often took decades for science to catch up to science fiction. This happened to Bayeswatch in just 5 years. AI progress is accelerating, so if I wrote a new story like this, it would get timelocked even faster.

Maybe I should title it Leviathan….

Discuss

If you’re anything like me, you may have a lot of notes from various introspective activities – annual / monthly reviews, worksheets, therapy notes, etc. Often these notes are just gathering dust in a paper folder or some forgotten corner of Google Drive. There may be some insights there – how personal problems change over time, what things you tried and what worked and didn’t, and so on. One use case for capable AI models is to sift through this data and extract patterns, insights and interesting tidbits that you may have forgotten or not noticed. 

Here are some things I got AI to do for this analysis:

• Transcribe handwritten worksheets and diagrams (I did have to take photos of the sheets – this was the most laborious part). The transcription was occasionally wrong and needed correction, but overall saved a lot of time. 
• Summarize my answers to “Hamming questions” over the years. This was originally a CFAR worksheet named after Richard Hamming’s question “What are the most important problems in your field?” but applied to your life. 
• In the past I have found it useful to make a visual representation of my problems as a graph showing how they contribute to each other. I had only a few of these from past years, so I asked AI to draw diagrams for each year based on my notes and worksheets.
• Compare worksheets and diagrams from different years, identify persistent feedback loops, look how loops and key nodes change over time.
• Identify interventions that worked in the past that I could try again (therapy and coaching notes are particularly good to mine for this). 
• Do deep dives to analyze specific patterns. 

Claude Opus 5 and Fable did a pretty good job at this. Fable had a tendency to over-compress its findings, and I had to ask it to use simpler language. I also tried this 1-2 years ago, and I think the models are getting better at picking up non-obvious patterns.

Learnings about problems and patterns

Here are some patterns I found in my answers to Hamming questions over the past 10 years: 

• Rate-limiting step / bottleneck: sleep, tiredness, focus, anxiety, and confidence were recurring themes over the past 10 years. On the one hand I made progress on these issues, and on the other hand they appeared from new sources, so overall there is some improvement but they are still there. 
• Things I don’t feel “allowed to care about”: earlier years listed various things like having kids and mentoring others, which I eventually ended up doing. Creative pursuits also appeared on this list a lot (e.g. drawing or writing poetry) and I still don’t do much of this.
• Things to do to “advance the plot” in my life: more regular meditation, try different kinds of therapy, getting help / advice sooner, more delegation / babysitting. 
• Things I tend to do as a poor substitute for what I really need ("pica"): general theme of procrastination and being reactive (scrolling social media, gravitating to logistical tasks over more open-ended tasks or hard decisions, joining projects run by others). Most of these have improved with better reflection and prioritization systems, but the core issues remain.
• “My life is fine except X”: what came up as X has generally shifted from mostly mental problems (anxiety / overwhelm) to external / object level things (kids waking up at night, uncertain AI future etc). 

One thing I learned from this is that some of my persistent problems appeared in different guises or with different causes. These days I mostly attribute tiredness, lack of sleep and feeling overwhelmed to the demands of parenting, but looking back at my notes from 2015-17 this was just as much of an issue (if not more) before I had kids. There are internal and external factors affecting sleep, and the external factors have increased after having kids, but the internal factors are pretty stable and the same habits are still useful (e.g. put away screens and wind down after 10pm). 

Overwhelm was a recurring feeling triggered by different things (finishing my thesis, balancing research and outreach, taking care of a baby, etc) which seemed like a mental pattern that attached itself to life circumstances rather than a fact about those circumstances. I had a default response to having a lot of stuff going on with a sense of shame about not doing everything or comparing myself to others, and this made me feel overwhelmed (while the same circumstances did not feel as overwhelming when I gave myself permission to drop things). 

There were several persistent feedback loops between problems (which became weaker over time). 

• The judgment loop: underperform → judge myself → feel anxious → perform worse. This comes up most clearly at work, but there is also a parenting version: feel anxious →  be triggered / impatient with kids → shame / guilt → anxiety. 
• The pressure loop: feel behind → do lots of stuff compulsively → focus on small things and create more pressure → feel more behind. This has gotten better with explicit prioritization structures, but the compulsiveness pattern is still there, particularly when working on the computer (e.g. sometimes I find myself unnecessarily switching tabs). 
• The disconnection loop: feel stressed → ignore internal signals → lose touch with what I want → more stress. This seems mostly fixed and didn’t show up in recent years. 

Comparing my bottleneck diagram from 2015 with one from 2025, there are some nodes in common (“should” mindset, confidence, anxiety) and some things that did not persist (neglecting internal signals, working memory, excessive effort, insomnia), which mostly map to solved problems. There are some new kid-related nodes though (“stress with kids” and “reduced alone time”). 

2025 bottleneck diagram

2015 bottleneck diagram

Learnings about approaches and solutions

There were various things that worked in the past that would be good to try again or do more often:

• Reflection periods in a different physical location (library, park, coffee shop, etc).
• Aiming for a crappy draft (to counteract writers block).
• Noticing when I’m stuck with something and asking for help sooner. 
• Aversion factoring: noticing if I’m avoiding doing something I would like to do, writing down what is good about doing it, what feels bad / aversive about it, and any counterpoints to those objections. I often ended up doing the thing after doing some aversion factoring on it. (Aversion factoring is traditionally taught with diagrams, but I found it easier to do in list form.)
• Using anger / annoyance as an unblocker: noticing when I’m feeling stuck or tense and writing down what I feel annoyed about, which helps me to process emotions in the background and get into a more agentic mindset.

There were also things that I tried several times that didn’t work:

• Putting time blocks on the calendar for things I intend to do, without supporting systems to make it happen. For example, reading blocks on my work calendar often got ignored in favour of project work that felt more urgent, and “alone time” calendar blocks on weekends kept not happening. If there are mental or situational obstacles to doing the thing, a calendar reminder is not sufficient. 
• Closing comms channels (email / slack) to avoid distraction, e.g. during focus blocks. This didn’t alleviate the emotional reasons for getting distracted, and I often ended up going back to the comms channel to get some information I needed. What did work was to do focused work entirely on paper without a computer (reflecting in a notebook, reading a printed paper, etc). Deleting social media apps was also ineffective because I can open them in the browser (while using an app that prompts me to be intentional when I unlock the phone worked better).

I identified some meta-principles on things that tend to work for me:

• Without chain of thought, I am not a general intelligence. I need to write everything down: todo items (before doing them), my mental state before starting work, thoughts about research directions or papers I read, loopy / obsessive thoughts, decisions, factoring aversions, etc. 
• Habits that stick are usually attached to an existing routine: meditation after breakfast at work, leg stretches while using mouthwash before bed, doing regular exercise classes. One takeaway is that if I want creative pursuits to happen, I should sign up for a class in drawing or improv or something, and then I would actually do it. 
• Habits that rely on willpower don’t work. It’s good to look for physical solutions to problems (medications / supplements, nasal spray for colds, weighted blankets, focused work in a different place without electronics, etc). Habits that necessarily have a mental component (like avoiding sugar / snacks) only work if I get into a mental frame where I just don’t do this thing, rather than a willpower-sapping frame of trying to do less of it.

Overall, the benefits I got from doing this kind of analysis can be broken down in two ways: looking at problems or approaches, and whether what I did was successful or not. I think each of these categories can be worth a deep dive with some AI help. 

Successful

Unsuccessful

Problems

Noticing problems that I successfully fixed or improved and appreciating that.

Noticing problems that have recurred despite various interventions, and reflecting on why.

Approaches

Being reminded of things that worked before that I forgot about.

Identifying things that repeatedly didn’t work in the past and probably need a different approach.

Discuss

It seems to me that a lot of technical ai safety people haven't done their capabilities homework - and that's a shame! I'll try to illuminate here mainly with an example as to why I think people who care about safety should totally pay more attention to the trends and actively engage with them - the case for safe AI not through an additional loss term but as a consequence of the learning algorithm!

RLVR

It's now been 1.5 years since R1 came out - the paper which really introduced RLVR (RL with verifiable rewards) through GRPO at scale. GRPO is stupidly simple, reminding of early REINFORCE algorithms: sample n traces, assign them a reward and make the advantage a normalized version of their reward, applied to the whole trace. In other words: for a trace which resulted in a correct final answer, slightly increase the probability of sampling each token of its trace and vice versa.

This is also what safety focused people generally engage with - and that's totally fair! While GRPO has gone through some variations since then (Dr. GRPO, DAPO, ...), these are mostly minor improvements that you should not waste your time on.

I further agree that this form of training is incredibly dangerous - we seem to now be reaching the amount of post-training required to meaningfully differ from the benign prior, and it's not exactly looking peachy.

Yet, it should be clear that GRPO also seems comically terrible from a capabilities standpoint. So has the capabilities-front been cooking up anything that might interest safety researchers as well? The answer is yes!

On-Policy Self-Distillation

I will keep this at a high-level:

• we would like a richer signal than 'increase probability of literally every token': we want nuance that really punishes bad tokens and rewards rare breakthrough tokens
• distillation offers exactly such a signal, but (a) we don't have a teacher model and (b) distillation is off-policy
• if we don't have a teacher model, we need something which can imitate it: we use the same (frozen) model but give it more information in the prompt, like hints, common pitfalls or a solution - now it can emulate a more capable model!
• there is nothing keeping us from just doing distillation on-policy

Let's walk through this with an example:

We have some math problem, say "What's 32 * 5?". We generate a completion, which turns out to be wrong:

Let's think about this step by step. [...] 30 * 5 + 2 * 5 = 140 + 10 = 150. Therefore the final answer is 150.

Notice that 30 * 5 is not 140, but everything else was perfectly reasonable arithmetic. Standard GRPO would now lower the probability of sampling all of these tokens even though there is a single error - everything else is fine and shouldn't be disturbed!

Now let's walk through On-Policy Self-Distillation (OPSD):

We can reuse the same student completion as example; after all, we are now also On-Policy. But this time we don't have some kind of binary reward for the entire trace - instead we create a second prompt for our 'fake' teacher model:

Solve the following problem: What's 32 * 5?
If you get stuck, you may use the following example solution as help, but first try it yourself:
32 * 5 = (30 + 2) * 5 = 30 * 5 + 2 * 5 = 150 + 10 = 160. Final answer: 160

Now, we aren't doing normal off-policy distillation, so we won't be using this prompt to actually generate anything. Instead we concatenate the student's attempt below!

Solve the following problem: What's 32 * 5?
If you get stuck, you may use the following example solution as help, but first try it yourself:
32 * 5 = (30 + 2) * 5 = 30 * 5 + 2 * 5 = 150 + 10 = 160. Final answer: 160

Let's think about this step by step. [...] 30 * 5 + 2 * 5 = 140 + 10 = 150. Therefore the final answer is 150.

Finally we do distillation over these generated student tokens: the context window with extra information acting as teacher, and the one without acting as student. The teacher will recognize that up to "30 * 5 + 2 * 5 =" everything is reasonable and not meaningfully disturb the student. Right before the '140' token, their predictions will massively diverge: even if the teacher is the same model, which might very well make this mistake, it now knows for a fact that it isn't correct. Down the line, say right before 'Therefore', it might also start trying to reverse the error, predicting tokens like 'Wait'/'But' in an attempt to correct the last step.

I hope the elegance of this approach becomes clear. It should remind you of how humans think about their past mistakes: with your new knowledge of what actually turned out to be a more worthwhile solution, you revisit your previous steps and consider where you went down the wrong path and should have explored other directions.

Safety

Between RLVR and OPSD - which would you prefer to be used to train frontier models? Assume that in raw intelligence, they would be the same.

I hope by now you shouted 'On-Policy Self-Distillation', even if only in your head:

• we should expect less overselling of work, one of the currently most severe faults in models
  • there is no LLM-judge (and corresponding reward) that the student could even oversell its work to!
  • if the student oversells its incorrect work for whatever reason anyway, the teacher will most likely know and actively push against it, trying to steer back in the right direction
• in general, less verifiable domains (like most of safety) actively get signal to not get left behind by increases in capabilities
  • if we include the target behavior into our example solutions or even just explain it, the teacher model will try to follow it, and the student will be steered towards it: from writing better stories (by giving a good example story) to adhering to specific safety notions
  • again, this really can't be overstated: there simply isn't a reward or any reward pressure anymore; instead we have a normal pre RLVR LLM which has a pretty good model of the fuzzy ethics of humans and just learns to push towards whatever we specify in our teacher prompt, including these fuzzy ethics!

Do I think this approach is perfect? No.

Do I think it's 100x safer than normal RLVR? Yes, GOD YES.

Empirical

So where does this approach currently stand - can we imagine it taking over RLVR soon?

I described in early December of last year the vision for precisely this algorithm (and more). In late January, a paper first appeared on arxiv which seems to have independently discovered & implemented it and the results were promising. Since then, there have been quite a lot of papers on it, many different variations (of which, sadly, a lot are slop). Nevertheless, we seem to be making progress.

In my post from December I already laid out the problem of 'Impossible Knowledge'. By this I refer to the fact that our teacher could just directly output "160", simply because it already knows stuff it shouldn't know yet. Teaching the student to imitate this 'Impossible Knowledge' will be pointless and not help capabilities once this knowledge is gone. I proposed an initial solution, and it seems to have been tried out just 2 weeks ago: the empirical results are too short for me to make clear statements, but initial results do seem promising.

In terms of frontier labs adopting it: it seems Cursor is actively using it for training its in-house models with the caveat that they are using both RLVR and OPSD - which mostly ruins the point from a safety perspective.

Conclusion

There is much slop and many incremental results in research of LLM capabilities - I do not think technical AI Safety researchers should waste their time on such results. But, from time to time, there are meaningful new directions and AI Safety researchers can NOT be the last to find out.

Just a few days ago, it seems to have come as a shock that Kimi K3 uses a residual stream which has an additional routing mechanism, complicating many interpretability approaches. Some people even initially believed that this was a good thing for the mech interp faction!

This kind of routing is NOT new and has been proposed and tested before, even at meaningful scale. Kimi K3 was simply the first real frontier-model (in terms of open-weights) to adopt it. I would have wished that especially mech interp researchers were more aware of such approaches because it's not unlikely that at least one frontier lab is by now using something alike internally.

But even aside from the more depressing side of labs chasing capability at almost any cost and safety researchers having to always be a step ahead as a result - being on top of capability research can also provide more optimistic sides, as I hope the example in this post could show.

Discuss

Teilhard de Chardin

This is a crosspost from my subtack.

In a recent article I discussed Robert Wright’s new book. His general thesis is that we ought to frame artificial intelligence cosmically, as a kind of evolutionary phase shift that will end up turning our civilization into some kind of global brain. We just have to decide on the kind of global brain we want to have.

I like Wright’s framing of the issue and I even sympathize with his invocation of Teilhard de Chardin, the maverick priest who started predicting the evolution of a “brain of brains” at the beginning of the 20th century. It seems evident to me that AI is just the pinnacle of a kind of process that has been going on for a long time, a process of ever greater complexification, cooperation and integration of living things and societies. If you think this is in any way plausible it throws up the question of the operand mechanism. Why does biological and cultural evolution show this tendency?

Teilhard de Chardin’s vitalistic answer to this question seems to be a poor fit with modern science. We naturalist Teilhardians need to come up with a better one! And here is where I start disagreeing with Wright: In his view, non-zero-sum dynamics, in conjunction with technological evolution, generates the vector of complexification. In my mentioned article I discuss in somewhat more detail why I think this does not work, but the general idea is that while non-zero-sum dynamics indeed occur, this by itself is not an explanation. It is just a formalization of conditions under which cooperation is helpful; it doesn’t explain why such conditions are becoming more frequent in the course of evolution in the first place.

My alternative explanation for why complexification occurs is metaevolution — the evolution of evolution. In the course of evolution, the medium of evolution itself evolves. The most obvious example of this is the emergence of memetic evolution next to genetic evolution. Memes are around because they allow us to adapt more efficiently as they are faster and more versatile replicators.

However, and now we come to the point of this post, I was kind of disappointed with the resulting argument because I could not give a nice and precise model for the metaevolutionary process going on. And while I still think that metaevolution is still waiting for proper general formalization, I think I can now give a first version of a simple computational model of the interaction between biological and cultural evolution in the emergence of ever more networked societies.

The general idea is that there are two kinds of replicator dynamics going on. Here are the assumptions (details can be gleaned from the repo):

• There are a certain number of creatures. We hold that number fixed for simplicity’s sake. At each iteration of the simulation, each creature interacts with a certain number of other creatures.
• Each creature has a preferred number of interactions per round. These are determined biologically (think prosocial instincts) and culturally (think social norms of cooperation like, of course, Zeus’ law). Each creature carries 100 binary genes and 100 binary memes. The preferred number of interactions of a creature is the sum over its genes and memes. So if you have 10 prosocial genes and 5 prosocial memes switched on, your preferred number of interactions is 15. All replicators are set to 0 at the outset.
• Each round we pair creatures such that each one, as far as possible, has the number of interactions that it wants to have.
• We assume that having 20 interactions is biologically optimal and generates the highest fitness. Imagine for instance, that less interactions means insufficient social support and more interactions tends to make social relations hard to manage.

Before introducing memetic dynamics, let’s have a brief look at the dynamics. As expected natural selection up close to the optimum of 20.

Now we stipulate dynamics for cultural evolution:

• Memes have a very low mutation rate and can randomly flip. But they can also spread by contagion: Whenever two creatures interact each gets one meme from the other, chosen at random.
• Memes spread faster than genes. For every 10 memetic update cycles, there is one genetic one.

We can now rerun the experiment:

Once an equilibrium is reached we see that the number of interactions happening is almost double of what it was before. Why? Because the memes that are prosocial are conducive to their own proliferation!

The simulation captures an instance of metaevolution because we are dealing with a conflict of two different orders of evolution. Genes and memes are in conflict here. While memes can never drive genes to extinction, because without biology there is no culture, it is still the case that memetic evolution leads to far more social interactions than would be biologically desirable. Of course, cultural evolution must still give some fitness payoff in biological terms, otherwise natural selection would just tend to favor creatures that are incapable of any kind of culture. Still, the model nicely illustrates that different orders of evolution can be in conflict, or at least tension, with each other.

A slightly unintuitive side effect of this is that cultural evolution is likely to make us less genetically disposed towards prosocial behavior. In the image on the right you see that the genetic preference is close to zero now because biological evolution tries to mitigate the fitness-decreasing tendencies of cultural evolution.

I suspect this kind of model is a far better intuition pump for grasping civilizational integration than mere non-zero-sum dynamics.

Subscribe to my subtack

Discuss

If you spend time looking at frameworks in the therapy/meditation/self-help space, you’ll soon find lots of conflicting claims about The One Approach for solving your problems.

In the therapy space, there are lots of models that say something like “all emotional problems are caused by trauma/suppressed negative emotions”. Then they disagree about how to deal with it.

Internal Family Systems (IFS) therapy holds that you should always move toward the suppressed material cautiously, explicitly checking in with all of the mind’s existing defenses to make sure that it’s okay to proceed. When you reach it, you should work with it in an “unblended” form, where you can witness the memory from the outside and offer it comfort and safety. For instance, if your parents used to yell at you, you can see yourself as a child whose parents were yelling at them, and then step in as an adult who comes to comfort that child and tell them that they’re okay.

Meanwhile, some psychedelic therapies essentially operate by temporarily blowing up all the defenses and throwing you right inside the original trauma. For instance, you might drink ayahuasca and viscerally feel the nausea and agony of having your parents yell at you, and then experience that trauma literally leaving your body as you throw it up. People in these kinds of circles might tell you that while the experience is horrible, it’s the only way to get to the root of things.

Some of the trauma-focused therapies say that it is crucial to develop an improved understanding of what the original trauma was and what it meant, to correct any incorrect beliefs it caused you to acquire. For instance, maybe you will go from “my parents always yelled at me because I was uniquely bad” to “my parents always yelled at me because they were under a lot of stress and didn’t have the resources to deal with it better than that”.

Others say that you don’t necessarily even need to understand what the trauma was - it’s enough to release the negative emotions associated with it. For instance, if the belief “I’m uniquely bad” involves a bodily feeling of nausea, you can find a way to release the nausea and stop feeling it. Then the belief becomes just an arbitrary sentence in your head that will no longer feel compelling.

Meanwhile, something closer to Cognitive Behavioral Therapy (CBT) might think that what matters are cognitive distortions operating in the present. It might hold that if you say “I’m uniquely bad so everything I do will fail”, then that’s a combination of labeling (turning your past failures into an identity claim) and overgeneralization (thinking that you fail at everything because you’ve failed at some things). It might then recommend something like tracking your actual successes and failures in order to notice that the belief isn’t true.

Also meanwhile, a solution-focused therapy might do none of this and just ask a question like “what’s a time when you felt more optimistic about succeeding at things”, and get you to remember times when you weren’t so burdened by your pessimistic beliefs. Maybe there was a time when you were in a group of people who were all excited and enthusiastic about something, and that got you so psyched up that you forgot all about your pessimism. Then what actually matters is surrounding yourself with the right people.

And while we are on the topic of external fixes, you will also find people saying that

• Actually, your problems are all caused by issues with your nutrition
• Actually, your problems are all caused by insufficient exercise (“just lift, bro”)
• Actually, your problems are all fixable with the right medication

Additionally, some meditation teachers will tell you that all of this is targeting the wrong things. You are only experiencing all of this as a serious problem because your mind has incorrect beliefs about what it is, and how tanha[1] causes you unnecessary suffering. The key is doing something like vipassana practice to get your mind to see what’s really going on, so that it will let go of the processes where it repeatedly hurts itself. Then both feeling bad about yourself and failing to achieve anything because of that turn into just sensations that you can witness without feeling hurt by them.

So what’s going on?

Obviously not all of these can be simultaneously true, so why do people believe that they are?

There is an obvious cynical explanation - “because that helps them sell their method better”. That is no doubt sometimes the case, but lots of these people seem sincere in their belief.

I think that for each of these approaches, there’s some particular shape of problem that the given approach really is either the best one or just very good for. You are only going to treat nutrient deficiency by fixing your diet; you are only going to solve low-level tanha by training your mind to closely observe it; some kind of trauma processing is good for PTSD. They also make different choices when it comes to values - something like the Internal Family Systems approach of dealing with trauma very cautiously is less risky and less unpleasant for the client than a system that throws the person right inside the trauma.

I have a more detailed model of psychology that tries to reconcile all of these into a single framework, but for this post, the main implication of it is just “minds are complicated and so are people’s issues; some issues need a specific approach while others can be affected by several”.

Issues in conventional medicine are similar. If you have a bacterial infection, you’ll just want to take an antibiotic. Meanwhile, high blood pressure can have multiple causes and also multiple interventions: getting more exercise, reducing one’s sodium intake, lowering stress, losing weight, taking one of various drugs, etc.

Likewise in mental health, a phobia with a clear specific target (like dogs or spiders) tends to respond well to exposure therapy. While depression may have various causes and contributing factors from biological to situational, and also many different interventions: antidepressants, different kinds of therapy, exercise, material improvements to one’s situation, social connection, sleep adjustment, etc.

This means that each individual framework can sometimes be either the one for a particular kind of problem, or it might be sufficiently general (like “get more exercise” is for health) that it makes sense for people to promote it in particular. Or, it might just happen to make a particular set of tradeoffs that happen to work especially well for some people, analogous to how ways of reducing blood pressure vary in how feasible they are to different people.

I’ll say more about that later, because I first need to deal with one obvious objection.

What I’ve said holds that different kinds of therapies and spiritual practices will have different effects and mechanisms of action, which is a more disputed claim. In particular, there is the well-known “Dodo bird verdict”, which holds that all therapies are roughly equally effective. Related, common factors theory holds that therapy effectiveness is mostly explained by things like the degree of empathy and therapeutic alliance in the process, rather than by the details of the approach.

So why do I hold my model?

It’s first worth noting that the “common factors are all that matters” interpretation is by itself pretty extreme. A 2017 review paper[2] recounts that

A highly cited 2002 review of 17 meta-analyses comparing active treatments with each other showed small, non-significant differences in outcome, which diminished further after the substantial effects of researcher allegiance were controlled for. However, these and similar findings have been questioned by those of other meta-analyses. Tolin, for example, reported that CBT was associated with a significant advantage over other therapies, at least in patients with anxiety or depressive disorders. Similarly, Hofmann and colleagues showed evidence for treatment specificity in a review of 269 meta-analytic studies examining CBT for different psychological problems. [...]

The common factor and specific factor positions might be less divergent than they at first appear. First, most specific-factor theorists concede that common factors are important. [...]

Second, common factor theorists seem to be increasingly tightening their definition of bona fide therapy. They suggest that the treatments must be delivered by trained therapists, based on psychological principles, and credible to the patients receiving them.

Third, most common factor theorists concede that some specific techniques are more effective than others for particular conditions. A major concession is that exposure-based procedures could be necessary for some types of anxiety disorders, although how exposure is performed might not matter. Both camps acknowledge that moderator factors, such as age, could be important [...] Other examples for which there is broad agreement for problem-specific effects include treatments for severe motor tics or cognitive deficits in schizophrenia.

There are also various methodological problems with running randomized controlled trials on therapy, which that same review discusses. So it’s not quite the case that the Dodo bird verdict would be the scientific consensus: rather its validity and scope is still an open question, and there is rough agreement that at least some interventions do work better on some cases than others.

Those aren’t the main reasons for why I have my own complex model of what’s going on, though.

My main reason is just that I’ve done various forms of therapy, meditation, and self-help, and they seem to clearly have different effects and work in different ways. Recognizing something as a cognitive distortion à la CBT, healing an exile à la IFS, or seeing subtle tanha à la insight practice are very different kinds of experiences, cognitively and emotionally. And while introspection isn’t a reliable guide to what’s happening in the brain, for me to believe in “these are all secretly the same” would be getting into “maybe there’s an evil demon messing with my mind and rewriting all of my experience according to its whims” territory.

When I got - after significant initial skepticism - into Internal Family Systems therapy and got training in it, I found it personally valuable and then started facilitating it to friends who were interested. Some it didn’t do much for, but some would say things like “this one session felt more useful than all my previous experiences with therapists combined”. That encouraged me to start doing IFS-based emotional coaching, where the results have been similarly varied.

Some of my clients get very little benefit from IFS, some quite a bit, and then occasionally there are ones who’ll show up for just a few sessions and find their life so drastically improved that they don’t feel a need to come back. I had one client who felt stuck in their life, had about three sessions with me, and then a year later e-mailed me to share a list of significant achievements and life improvements that had happened since then that they attributed to our work - when none of the previous therapists and coaches they’d worked with had felt helpful, and they’d previously felt skeptical of the possibility of ‘discrete step’ progress.

This kind of an outcome is definitely not the median outcome, but it’s also sufficiently common that I’m not surprised by it. And yes, there is regression to the mean and people do sometimes improve spontaneously… but it doesn’t really feel plausible to me that a client would have the experience of a breakthrough session and then just totally coincidentally happen to improve just then on something they’ve been struggling with for years. Maybe if I was seeing a hundred clients per year, you might expect something like that to happen by pure chance, but “some dozens” on a busy year is more accurate. And I’ve usually gotten a few of these per year.[3]

Notably, “some clients benefit a lot, some not at all, and the rest are in between” is entirely compatible with the Dodo bird verdict. If you take an average of the effect over all clients, you’d probably get something like “IFS works roughly the same as every other therapy”. (Indeed, the small number of randomized studies we do have on IFS say that it works roughly as well as whatever it is being compared with.) Standard parallel-group RCTs cannot estimate the extent to which any given person would have responded better to another treatment, because people are randomized into only one treatment arm.

The simplest explanation to me would be that for whatever reason, some interventions just work better on some people than others, in ways that are related to how their individual psychology works.

You might notice that I’m treating “specific interventions work best on specific issues” and “specific interventions work best on specific people” somewhat interchangeably. I started by suggesting that - say - IFS works best on particular kinds of problems, but now I’m talking about what kinds of people IFS works best for.

This is because I think these can’t always be cleanly distinguished. The review that I quoted above mentioned a finding that Acceptance and Commitment Therapy might be better than CBT for older adults with chronic pain, “perhaps because the therapeutic focus was more aligned with learning to live well with pain, rather than the treatment being expected to reduce the pain.” I didn’t want to cite this as evidence because the empirical evidence for it looked very weak at this stage, but let’s consider what it would mean if this was true.

The review called this a “moderator factor” for the general issue of chronic pain. But it seems slightly questionable to me whether the kind of permanent chronic pain that you might get from living to an old age should be considered the “same” issue as chronic pain you’re expected to eventually recover from! Your orientation to something you think might go away and something that you expect to just worsen is entirely different.

Likewise, people vary quite a bit in how they think and process the world. Some people have a constant internal monologue narrating everything that happens to them, and that have an “inner critic” in the form of a voice that is literally criticizing them and saying bad things about them. This is totally alien to me: unless I’m thinking about what to say in conversation or what to say in an essay like this one, most of my thinking is non-verbal and non-linguistic. My inner criticism doesn’t feel like anyone saying bad things about me, it just feels like I’m intrinsically bad.

Now suppose that I have a friend who does have an active inner monologue and a voice-like inner critic. If both me and my friend are feeling bad about something we’ve done, then on one level we’re having the same issue, but at the same time the form that the issue takes is entirely different. A therapeutic intervention along the lines of “ask the voice why it wants to criticize you so much” or “imagine the voice talking to you in a cartoon-like high pitch that makes it sound ridiculous” might be helpful for my friend, while making me just go “huh?”.

Now, this would imply that you could figure out which therapy works best for who, and then match clients to the therapies that work best for them. So far, studies that have attempted to do this have had disappointing results. But then, it’s unclear what good predictors would be.

If you asked me to give you an objectively measurable trait that could be used to predict whether someone benefits from IFS, I’d struggle. I sometimes tell people that “I tend to get the best results with people who’ve already tried something like Focusing or IFS and found it promising”. But that’s not really a predictor, it’s just saying that my method works best for the kinds of people who have already found it to work for them.[4]

Being intuitive and in touch with your emotions helps quite a bit, but I’d expect such people to do well in any form of therapy. You might also expect something like “finds the notion of the mind being divided into parts intuitive” to be a good predictor… but I initially kept bouncing off IFS because all descriptions of it sounded hopelessly woo and unscientific, while some of my clients totally buy the model intellectually but still struggle to connect with their parts.

So it seems to me that different people’s problems are configured in different ways, some therapies are better than others in solving a particular problem or configuration but none is universally best for everything, and it’s hard to predict in advance what would work for any given person.

Some of the therapies and approaches might also change one’s configuration so as to unlock new approaches. For example, IFS and many of the other experiential therapies are often not very effective for people who have difficulty feeling their emotions. Practicing Gendlin’s Focusing and doing body scan practices are approaches that can get people more in contact with their emotions. This can then make it easier to get benefits from other therapies.

Complexity of problems as source of hope

I noted before that many frameworks have their own ideas of what works and what needs to happen for things to get better. If you accept any one of these claims as truth, it can be strongly motivating - “I just need to heal my trauma to fix my problems” or “I just need to surround myself with enthusiastic people to fix my problems” - as long as you seem to be on track to doing that. But if it starts looking like you’re not making progress toward it for whatever reason - maybe you can’t find the original trauma, maybe you’re failing to find sufficiently enthusiastic people - then it is easy to start feeling hopeless. Alternatively, you might retain the excitement for the method, and keep grinding on something that isn’t actually a good approach for your problem.

But if there are many different interventions that could potentially work, then that’s a source of hope. If you have been grinding on one school’s approach for a while and making little progress, it’s still possible that another one will work! And while it might not solve your problems entirely, it will still make your life easier and improve your odds of tackling the remaining parts of it - or maybe just conclude that the problem is sufficiently under control that you can let it be and move on.

So why do so many people think their framework is “The One”?

With all of that out of the way, I can now finally discuss possible reasons for practitioners to think that their framework is The One True One. Now many of these might affect me as well - and have historically affected me, for there was a time when I thought everyone just needed to learn IFS and then all of our psychological problems would be fixed. Turns out that I still think it’s great and extremely beneficial for some people, but alas, it didn’t fix even all of my psychological problems.

Typical mind fallacy. It is a common experience for people to hear different kinds of advice, note that it doesn’t work for them, and then assume that the advice was bad rather than the advice being something that just didn’t work for them. Scott Alexander writes:

I only really discovered this in my last job as a school teacher. There’s a lot of data on teaching methods that students enjoy and learn from. I had some of these methods...inflicted...on me during my school days, and I had no intention of abusing my own students in the same way. And when I tried the sorts of really creative stuff I would have loved as a student...it fell completely flat. What ended up working? Something pretty close to the teaching methods I’d hated as a kid. Oh. Well. Now I know why people use them so much. And here I’d gone through life thinking my teachers were just inexplicably bad at what they did, never figuring out that I was just the odd outlier who couldn’t be reached by this sort of stuff.

If it is the case that different therapies work for different people, then someone may have gone through a number of therapies before finding one that clicked for them. Maybe CBT didn’t do anything for them but IFS did, or vice versa. Then they find a community of people who have all had a similar experience, end up mostly talking to each other, and conclude that “obviously IFS is the only thing that works while all the people promoting CBT are just inexplicably bad at doing therapy”.

The ontological “when all you have is a hammer” issue. These frameworks do more than just offer possible solutions. They also have their own ontologies about what the problem even is. Internal Family Systems sees everything in terms of parts of the mind; some meditative traditions see everything in terms of tanha; the biological frameworks might frame things in terms like inflammation, cortisol, blood sugar, and sleep debt. Since a lot of phenomena really are explained and well-modeled through these kinds of concepts, it’s possible to start seeing everything through a particular framework’s vocabulary, which makes it harder to conceive of any alternatives.

This is especially so as getting better at applying each framework tends to involve learning to see the world through its lens. Learning IFS has you seeing everything in the mind as parts, learning CBT has you recognizing cognitive distortions in all of your issues, vipassana might have you seeing tanha everywhere. Each of these operates on a somewhat different level of perception, so that focusing on one level may blur out the others.

Selection effects are a thing. It’s easy for various selection effects happen - for one, the practitioner has to market themselves somehow, and any kind of marketing will be appealing to some people and unappealing to others. People will also self-select - someone who thinks their problems are caused by trauma is likely to be looking for a trauma specialist. This might make it more likely for practitioners to get the kinds of clients their framework is good for.

Also, someone who thinks their problems are caused by trauma is also likely to believe in frameworks that emphasize the significance of trauma in particular. That means that the client is less likely to challenge the practitioner’s beliefs and more likely to actively affirm them. Logically, “your clients agree with you about your framework being valid” shouldn’t cause a practitioner to change their beliefs - the practitioner is supposed to be the expert, and they should be looking at results not pre-existing client beliefs. But practitioners are humans too, and humans tend to automatically believe more in anything that people in their social environment affirm.

Wishful thinking. “I know the one method that is key to solving everyone’s problems” can be a belief you are very motivated to hold - either for an egoistic or altruistic reason! The thought of being able to personally fix everyone’s problems can feel very satisfying to imagine. And if one is strongly motivated to help other people, they may want to believe that there exists a way to fix everyone’s problems and that nobody is beyond saving.

Many problems are multicausal. Maybe you have ADHD, trauma, and you don’t get enough exercise. Each of these is likely to hurt your emotion regulation, and the effect of the three may be closer to multiplicative than additive. So if someone believes that “you should just lift, bro”, even if the lack of lifting wasn’t the only core reason for your mood problems, following the advice may still produce a sufficient improvement to make one think that it was the core problem.

Multicausality may look monocausal in retrospect. I mentioned earlier that many practitioners may believe in a method because it turned out to be transformative for them. This may be true even if they actually needed a lot of other methods to get there.

Many people working through their issues will experiment with many different methods, each method allowing them to dissolve a part of the problem. Then they find some particular method that lets them solve the rest of it… and then attribute all of their success to that method, thinking that if they had just used that method all along, they wouldn’t have needed to waste all this time.

But all of those previous methods weren’t actually wasting time. They were the right methods for the stage the person was at previously, and the last method they found might even have been entirely ineffective at that point.[5]

Persistence often pays off... Some issues that are caused by e.g. trauma don’t seem obviously connected to anything traumatic at first. And some problems that seem obviously psychological may turn out to be physical with sufficient investigation. Some very deeply buried trauma or obscure medical condition may require a lot of looking into by expert practitioners before the underlying trauma or origin becomes apparent, and then one might get significant results.

This means that if someone has a strong frame of “everything is about trauma” or “everything is physiological”, they will keep pursuing that hypothesis at a point where someone holding broader views would already have switched to another hypothesis - which will be the correct thing to do in cases where someone’s problems are mostly caused by that, and wrong in cases where they aren’t.

The trauma specialist who keeps digging for a year before the buried trauma surfaces may be behaviorally identical to the trauma specialist who keeps digging for a year on someone whose problem was a thyroid condition. Whether that digging was correct or not depends on the base rate of trauma-caused depression versus thyroid-caused depression - which is exactly the thing that’s in dispute between the different practitioners.

…even if the results aren’t impressive. With enough digging, one might always be able to find some trauma that could be related to the issue, or some medical issue that could conceivably be making things worse. This may be true even when it’s not actually a particularly crucial part of the issue. So a person who has repeatedly had the experience of, say, “enough digging will surface an underlying trauma” may find their “everything is caused by trauma” model confirmed once they do manage to find one.

If this then fails to significantly shift the issue, when fixing many other traumas has significantly shifted people’s issues, the failure may be attributed to something like “this trauma must have other aspects we haven’t managed to uncover yet”. (Which may still sometimes be true and key to resolving the issue!)

Therapeutic progress is intrinsically noisy. Some clients will inevitably drop out or fail to make progress for reasons entirely unrelated to the validity of the framework. Maybe the client’s and practitioner’s personalities just didn’t happen to work well together, maybe the circumstances in the client’s life got a lot more stressful, maybe they just decided to focus on something else. It’s often the correct move to expect this to happen and to not trust the framework any less for it.

But this means that it becomes hard to distinguish between “all the clients who stuck with me long enough saw significant improvements because the method happened to work for them and made them persist” and “all the clients who stuck with me long enough saw significant improvements, but couldn’t stick to it because of factors outside my control”. This doesn’t necessarily even take the form of blaming the client - the practitioner can even put it down to something like “I wasn’t good enough at motivating them”.

Combined with the multicausality, the practitioner might even have witnessed that the clients who dropped out saw genuine progress. The “just lift, bro” guy might have seen someone’s emotional regulation make real progress because of the lifting - but failed to note e.g. the client’s trauma or ADHD making it too hard to persist with the weightlifting in the long term. So it will look to them like “if the client had just continued, they’d have felt much better”... which may be true, but is failing to notice the thing that prevented the client from “just continuing”.

Frameworks often have different values. Suppose that you are in a high-achieving career, which you are pursuing due to an internalized belief that you absolutely need to be high status in order to be loved. This leads to some distorted thinking, so you work with a CBT therapist to fix it. This leaves your underlying emotional compulsion to pursue status at the expense of everything else unchanged, but frees you to pursue it more effectively. It also makes it so that you are less likely to burn out, which would allow you to work on your deeper issue.

Is this a success?

Even if one was trying to rigorously measure the success of various frameworks with randomized controlled trials - and was able to control for all the confounders - the possible weaknesses of this approach will be invisible. No RCT measures a counterfactual like “would the person have been ultimately happier and more fulfilled if they had been allowed to burn out and then pursued an entirely different methodology”. Instead, it might just show that your mental health has improved and that you are achieving all of your goals better.

But other practitioners - maybe ones who had themselves burned out, been forced to do deeper work, and then found themselves grateful for that - might see this result as worse than doing nothing. (Note that I’m not claiming that pursuing a high-achieving career is always done out of unhealthy motivations. Just that it sometimes is.)

The most extreme disagreement might come from the Buddhist-leaning teachers. Some of them might hold that any life motivated by the belief that one needs to chase positive feelings and avoid negative ones is a prison and a delusion.

Meanwhile, some of their approaches might lead people to start preferring an existence of just meditating in a monastery for the rest of their lives… which the more engagement-focused practitioners might see as a failure to promote a genuinely flourishing life.

And as I noted before, value differences also covers things like differing tolerance for risk. The IFS approach of the “always proceed slowly, only work with material that you can witness from the outside” minimizes in-session discomfort and the risk of the client getting overwhelmed. A psychedelic intervention that throws the client right back into their original trauma to release it can be faster and deeper, while also being riskier and much more painful.

How to find the help you need?

Okay, so it’s nice to know what makes everyone think they have The One Solution. But you don’t just want to theoretically understand it, you have some issue that needs fixing, and you want to find a practitioner who’ll actually help you. What do you do?

Alas, I don’t have any clear-cut solution. But here are some thoughts:

• Most obviously, be willing to try different things.
• Also, be willing to try different practitioners! Even if the framework does make a difference, common factors and client-practitioner fit also matter enormously. I personally like IFS but I’ve also heard horror stories about bad IFS practitioners, and I’m sure there are lots of people who’d benefit from IFS who I just wouldn’t happen to click well with.
• Do a common-sense sweep to your issues. If you’re never getting any exercise and only eating frozen pizza, you don’t need to see an expert to know there are potential gains to be made. (That said, if awareness of never getting any exercise and only eating frozen pizza fills you with such shame that you can’t make any progress toward fixing this, do see a therapist about that and don’t feel like you have to fix the exercise and nutrition first.)
• If possible, start with cheap tests. Blood tests to exclude an obvious medical issue are lower cost than a year of trauma therapy.
• Ask the practitioner on their track record with similar problems, and when they would suggest you stop working with them. If they say this works for everything, be suspicious. Though a more caveated response like “this doesn’t work for everything, but I think letting go of tanha is the most valuable thing so I’m only focusing on that” shows self-awareness.
• Ask them about the last time when they did recommend someone stop working with them. If they cannot think of a single time, or have to pause for a long time to think of somebody, that suggests they do not have real off-ramps.
• In that regard, it’s worth being explicit about what your goals are and making sure yours agree with the practitioner’s.
• Do sometimes delegate things to the practitioner - if they say “I know you’re not feeling any progress right now but I honestly think we’re close” and they seem honest, probably give that a chance.
• There are different bottlenecks - if one approach fails to work for a while, it may work better later (e.g. some people find that CBT starts to work after trauma work; approaches focusing on bodily sensations may unlock other approaches bottlenecked on your ability to feel your body).
• If you have the time and energy for it, some things can be tried in parallel and may actually be synergistic. Combining antidepressants with psychotherapy is the classic one. Or if you have a lifelong aversion to exercise, you can see a doctor to see if there’s a contributing medical issue while also seeing a therapist about any emotional factors related to it. The downside of this is that if you try a combo and you feel better, you won’t necessarily know whether it was one or the other or both together that made the difference.

Finally, don’t forget that “I’m going to go through every possible therapy and spiritual practice until I find one that works to fix my problems” can also be a form of avoidance.

Gretta Duleba has an article about Relationship Known Issues. In software, a Known Issue is a problem that the developers know about but have no plans to fix - because it would be hard, because the issue doesn’t matter much, or because they just have more important things to do.

Relationships can also have issues. Maybe one partner snores so much the other can’t sleep, but the two would like to sleep in the same room anyway, and this is a source of ongoing pain. It then makes sense to pursue solutions, but if nothing works out, it might be best to just treat it as a Known Issue and live with it rather than start resenting the partner or the relationship for it.

One’s life can also have Known Issues. If nothing else, the inevitability of everyone’s death is a Known Issue. It’s good to look for solutions if it seems like solutions are available, but it can also be a relief to just give up on trying to find one.

Alas, it seems impossible to ever know for certain when you should give up on trying and when you should persist. “I will try everything you fix my issue” can be a defense, but so can “there’s no point in even trying anything”.

I’m treating that impossibility of knowing as a Known Issue with life.

1. ^

   Often misleadingly translated as “attachment” or “craving”.

2. ^

   Mulder, R., Murray, G., & Rucklidge, J. (2017). Common versus specific factors in psychotherapy: opening the black box. Lancet Psychiatry, 4(12), 953-962.

3. ^

   I can’t totally reject a pure common factors explanation on the basis of these examples alone, though. “This person had a breakthrough with me after trying several other therapists and coaches” could be driven by “my method worked particularly well for the person” or it could be driven by “personality-wise, I happened to be a better fit for them than anyone else they tried”. And e.g. the people who benefit the most from working with me also tend to be the ones I have the best rapport with, so I do certainly agree that the common factors are very important.

   I should also note that drastic improvement within a single session is not unique to IFS, but also happens in CBT as well as a range of other therapy modalities. My interpretation is that various modalities can happen to hit exactly the right spot for what the client needs, but a common factors proponent could also argue that this is a sign the modality doesn’t matter and it’s only about getting a good match with the therapist.

   To which I can only say that I see the argument, but having tried multiple modalities, that just doesn’t feel plausible to me.

4. ^

   The first rule of tautology club is the first rule of tautology club.

5. ^

   For something similar in a slightly different domain, see the Burrito Tutorial Fallacy.

Discuss