Сборщик RSS-лент

Nestor Guillen (like many, many others) wrote a blog post reflecting on recent progress in getting AI systems to prove results in pure mathematics: Happy those able to know the causes of things. It contains a nice analogy that may or may not be correct, which is also discussed with nicer pictures in another blog post reflecting on, etc., etc., by Álvaro Lozano-Robledo: What should we tell our students?.

Consider the entirety of truths of pure mathematics[1] as some sort of geometrical space. The truths so far proved by humans form a subspace, presumably in some handwavy sense a nice compact[2] one.

In many, many cases, proving a mathematical theorem is mostly a matter of combining clever ideas already had by other people. Actually carrying out the combination may still be difficult and/or impressive intellectual work, but any reasonable observer would say: well, the real contribution here is recognizing that you can combine the ideas of Smith, Jones and Ali so solve this problem. Guillen suggests that this is a bit like taking some points in that discovered-so-far subset of the space of mathematical truths, and adding a new point that's a convex combination of them: a point on the line joining them or the plane through them or whatever.

By repeatedly doing this you may arrive at a lot of new points, but there is a sense in which your progress will necessarily be limited: every point you can form in this way will, by definition, be part of the so-called convex hull of the set of points you started with, the set of all convex combinations of points in that set. You can think of the process as "filling in the holes[3]". Here are a couple of illustrative images[4] from Lozano's post:

Now, of course, what Guillen and Lozano are suggesting is that today's AI systems, even the best ones, are only "filling in the convex hull". They may possibly be suggesting that any plausible AI systems that work in basically the same way will also only ever be good at "filling in the convex hull". I think these are interesting hypotheses and worth thinking about some more. We could call the first the Weak Convex Hull Hypothesis and the second the Strong Convex Hull Hypothesis; in what follows I'll mostly be engaging in vague handwaving and will just talk about "the" Convex Hull Hypothesis.

Convex parrots

The CHH obviously somewhat resembles an earlier thing that people have said about LLMs, the Stochastic Parrot Hypothesis. (Wikipedia; the original paper by Gebru, Bender, McMillan-Major, and Mitchell.) Although one still hears people saying smugly that of course LLMs are just statistical text predictors that regurgitate a mashup of their input, I think this critique has become visibly less credible as today's AI systems have produced (among other things) proofs of mathematical theorems that hundreds of very gifted humans have laboured long and hard on but not been able to find. But the CHH version of this critique seems less blatantly at odds with reality, and it has some of the same implications as the SPH does.

Is the CHH true?

CHH seems kinda-plausible for two reasons. First, the thing at the core of today's AI systems really is a statistical token-predictor trained using a vast pile of human-written text. Second, it does seem like even the most impressive things these systems have done could reasonably be described as "filling in the convex hull". Third, it also seems like almost all human discovery/invention/creativity could also be described that way. (This last observation, if correct, makes CHH more plausible and simultaneously less damning.)

The first was pretty much also the argument for the SPH, which I consider thoroughly discredited by now. There are a couple of difficulties with its premises: LLMs are not purely trained on human-generated material any more (for instance, RLVR involves a large body of computer-generated training data, and RLVR is plausibly responsible for a lot of recent progress in LLM performance in software development and mathematics), nor is their training all about token-prediction (for instance, RLHF tries instead to train them to do things humans judge to be good, though I think much more of their training is token-prediction than anything like RLHF). And it's not so obvious that one can't build something with a really good token-predictor at its core that does genuinely new things. (Consider neural-network-based computer go players. They're trained to do something a bit like "predict the next move". But what's happening is that the neural network alone is being trained to predict what the neural net plus a load of tree-searching produces, and to whatever extent that works it produces something smarter than whatever you trained it on.) But, still, it feels like there's something to this argument. If you put vast effort into training something to reproduce its training data, you should expect what it does to at least rhyme with reproducing its training data.

The second is at least somewhat an empirical question, and actually I don't know for sure that it's true. But e.g. my understanding is that the Navier-Stokes proof, while very impressive and absolutely the sort of thing that would win a human mathematician the Fields Medal, is doing something that fits into a broad approach already laid down by human mathematicians. I would be very interested in the opinions of anyone who's looked deeply at the more impressive results in OpenAI's recent dump of hundreds of proofs.

The third is also somewhat an empirical question, and resolving it with real confidence would require examining everything ever created by humans. But I am fairly sure of the following things:

• It's commonly said that first-rate mathematicians, engineers, etc., typically have a small set of "tricks" they're extremely good at applying and at finding possible applications for, and their impressive output comes from repeated application of these.
• Advice on How To Be Creative often includes something along the lines of "try bringing disparate ideas together and see what comes out". (Arguably such advice is aimed at people who aren't very creative. On the other hand, I'm fairly sure at least some of it is obtained by looking at what actually-creative people do.)
• A lot of what's widely regarded as first-rate creative work doesn't involve going beyond the bounds of the sort of thing that's been done before. Bach's fugues and Beethoven's symphonies are wonderful pieces of music, they have plenty of good ideas in them, but people were writing fugues well before Bach and writing symphonies well before Beethoven (I think he was the first person to make the third movement have a scherzo rather than a minuet, but that hardly feels like Outside The Convex Hull).
• Things that feel like big advances often have smaller antecedents. Consider Einstein's special theory of relativity. This was a huge conceptual advance. Time as just one coordinate! Different notions of time for different observers! Time and space getting "mixed up" for observers travelling at different speeds! Well, yes, but also no. The idea of time as a fourth dimension had been around for decades, both in handwavy popular discourse and in actual mathematics. The equations describing how things get transformed at different speeds were due to Lorentz and Poincaré, though they weren't interpreted the way Einstein interpreted them. If we'd had AIs like today's back in the early 20th century and they'd got there before Einstein, I'm pretty sure today's critics would have had no hesitation about claiming that they were just remixing d'Alembert and Lagrange and Lorentz and Poincaré.

If it's true, does it matter?

Let me first of all make the obvious remark that in the doomiest scenarios we can imagine we have much worse problems than anything to do with stifling human creativity. So let's assume arguendo that those scenarios don't play out. (That's more likely if for some reason the AIs never get too much smarter than humans, so while making that assumption we should think that more likely than we otherwise would.)

We might reasonably worry about the following scenario:

• LLMs get incredibly good at filling in the convex hull.
• Creative humans generally get their start by finding good bits of convex hull to fill in.
• With LLMs doing this much more effectively than humans can, we have less motivation to do it ourselves, less opportunity to impress others by doing it or get paid for doing it, etc.
• Humans largely stop learning to be creative, and in particular much less often develop the ability to push beyond the convex hull.
• LLMs never get as good at extending the convex hull as humans were, and humans are doing it much less than they used to.
• The overall rate of expansion of knowledge and understanding decreases.

We might, a bit more disreputably, feel

• that if AI systems are only "filling in the convex hull" then we are still importantly superior to them, we have real creativity when they don't, there's still a need for us, yay humanity, boo AI; or
• that if AI systems are only "filling in the convex hull" then we have licence to be unimpressed by what they achieve; or
• that if AI systems are only "filling in the convex hull" then the real credit for what they do should go to the humans who provided the previous points they took a convex combination of, and the AI boosters who ask us to be impressed by the AIs are just trying to steal credit from those humans.

I'll deal with these first.

• There are certainly things we (or at least some of us) do better than today's AI systems. Humanity is not yet obsolete, whether or not it will be so one day. If you want to say "yay humanity" or "boo AI" about this, you should feel free to do so.
  • Of course we have no guarantee that this will remain true. Even if some highly abstract argument suggesting that nothing at all like today's AI systems will ever do more than filling in the convex hull, we already have good evidence that being very good at filling in the convex hull can achieve impressive things, and I would not want to bet that it can't in particular come up with better ways of building or training AI systems that make them more genuinely creative.
• You can be impressed or unimpressed by anything you want. The purpose of being impressed or not is (I guess) to adjust our expectations of whoever or whatever did the thing. If something shows its ability to prove mathematical theorems that top human mathematicians had trouble with, then I will expect it to go on being effective at doing that and other things sufficiently like it.
• When a human being puts together other human beings' ideas in an unexpected way and thereby achieves something no one had been able to do before, we typically give credit to the predecessors and to the innovator. I don't see any reason why we shouldn't have the same attitude to AI systems.

So, what about the nightmare scenario? I genuinely don't know. I think it depends on how much progress (of whatever sort we might care about) depends on "genuinely" new ideas that aren't "just" slightly perturbed convex combinations of existing ones. Which brings me to ...

A wrinkle: convex combinations plus noise

Slightly perturbed convex combinations, I just said. After all, when you bring someone else's ideas together to do something new, you are adding something. What happens to that repeated-convex-combinations model of (AI or human) progress if we allow for a bit of noise?

Here is a small set of starting points in 3d space[5], along with 10,000 points produced as follows: construct 100 random convex combinations of existing points on the convex hull of what we currently have[6], then add them to the set; repeat 100 times. Obviously this process never goes beyond the original convex hull.

Now suppose that when we construct each new point we add a bit of noise. Let's say spherically-symmetrical Gaussian noise, with standard deviation 1% of the length of the edges of that cube. Here's what we get:

The convex hull at the end of this process is now about 11% bigger in volume than the original convex hull. Make it 100k points rather than 10k and we get this:

That's 23% bigger than the original convex hull. I think the lopsidedness is a path-dependent thing: once you start getting more expansion on one side, more of the points on the convex hull are over there and you will keep sampling from them.

I took four points at a time for my convex combinations so that even without noise we could get points in the interior of the cube as well as on the outside. Using fewer means more of a tendency for new points to be on or near the convex hull, and hence more expansion. Here's what we get with 100k points, noise s.d. 1% of original side length, and taking just two at a time:

The scale is different here because the final convex hull is larger. This one is 3x the volume of the original.

Exactly how this process behaves depends on exactly how we select points to combine. If we do the same process that produced the last image, except that we pick points completely at random from the ones we have already, we get this, whose volume is only 9% higher than that of the original hull:

If we maintain a set of "particularly extreme" points in the convex hull, namely those furthest out in a somewhat-uniform set of 128 directions, and then mostly pick points from that set to combine two of, we get this, whose volume is about 6x that of the original convex hull.

I think it is obvious that any variant of this process will, if we give it long enough, end up filling all of space. (I don't claim that the same is true for the thing this is an analogy for.)

Tentative conclusion: if the filling-in-the-hull process introduces any novelty at all (which it seems hard to argue it can't, whether it's humans or LLMs doing it) then this process can in fact expand the bounds of knowledge beyond the original convex hull, perhaps by a lot. It seems pretty plausible to me that something like this is in fact the main way in which human knowledge and understanding have grown in convex-hull-expanding ways.

So, in particular, while the scenario laid out above would be bad for human creativity, and demoralizing, and arguably put too much power in the hands of whoever owns the best AIs, and so forth, I think it probably wouldn't put an end to the growth of knowledge and understanding.

1. ^

   You might prefer to replace "truths of pure mathematics" with something more concrete like "theorems of ZFC".

2. ^

   I definitely don't mean "compact" in the mathematical sense here, and in fact taking it that way would lead to another variation on the same analogy, but I think a less fruitful one.

3. ^

   "Concavities" might be a better word.

4. ^

   Picky mathematicians may notice that the thing drawn in the second image isn't actually the convex hull of the original blob, but I don't think that's deliberate -- though the difference has maybe a little connection with a point I'll be making later.

5. ^

   Actual idea-space is much higher-dimensional. One consequence of this (pointed out to me by Oscar Cunningham) is that if you start with something spiky and pass to its convex hull, the ratio of volumes of the two can plausibly be much larger in higher dimensions than in lower. (Though obviously it can be arbitrarily small in any dimension > 1.)

6. ^

   So that we don't end up with almost all our points clustered in the middle. This is somewhat representative of what actual scientists, engineers, artists, etc., do: the more excitingly innovative and creative something else is, the more it will get imitated and mined for ideas.

Discuss

Epistemic status: amused. Labels read, court records checked.

AI use: research (more examples and context), grammar and light phrasing.

We've already looked at marketing claims that are unnecessary because they are the default, in Gluten-free water. However, the landscape of weird marketing tricks in the US is far from exhausted, and today we will look at marketing mimicry:

Source: Walmart

Is this butter or not? Let's get closer and have a better look:

The same tub, up close.

Oh, it is not butter, it is "I Can't Believe It's Not Butter!". I remember seeing this product in a store myself and finding it extremely amusing and clever. At first I thought it was some variation of margarine. However, you won't believe it's not MARGARINE either! In the US margarine must be at least 80% fat, and this is a "buttery spread, 45% oils" whose first ingredient is water.[1]

Anyway, the claim is that whatever this is, you cannot distinguish it from butter. I find it funny that the logo may suggest to an inattentive customer that this is butter, while at the same time managing to say that it is not butter and that it is indistinguishable from butter. The number of layers of meaning here is impressive.

Froot Loops

Froot Loops, Mega Size. Source: Walmart

Does Froot Loops cereal contain fruit? It does not, only "natural fruit flavors". In 2007 the box also had small drawings of a lime, an orange slice, a lemon, cherries, a raspberry and a blueberry around the "natural fruit flavors" banner, and there was a court case, McKinnis v. Kellogg USA, on whether this was misleading.[2]The claim was that:

• Froot sounds like fruit.
• Brightly colored cereal rings suggest fruit.
• Illustrations of fruit surround "Natural fruit flavors", which is written in a disproportionately small font.

The court dismissed the case with prejudice. "Froot", it said, "might be a fanciful take on the word F-R-U-I-T", but it sits inside a trademark, not in the ingredient list: "No reasonable consumer would view the trademark 'FROOT LOOPS' name as describing the ingredients of the cereal."

The court also found that the rings "in no way resemble any currently known fruit" and, more usefully, that pictures of fruit on a box are not a claim that fruit is inside: FDA rules allow fruit illustrations to indicate a "characterizing flavor" even when the product contains nothing derived from that fruit (21 CFR 101.22(i)).

Bac'n Pieces

Bac'n Pieces. Source: Walmart

The situation here is a mess. So these are "Bac'n Pieces", which are BACON FLAVORED? This already strongly suggests that Bac'n is not bacon. Moreover, if you keep digging deeper, they are also artificially flavored, and they are bits. Can you guess what this is? (The answer "not bacon" is not accepted.) Walmart calls them "McCormick Kosher Imitation Bacon Bits", and the top ingredient is TEXTURIZED SOY FLOUR.

While researching this post I discovered that this design is legally required: if a food contains artificial flavor simulating the characterizing flavor, the name must be accompanied by the flavor name ("bacon flavored") and by "artificially flavored", each in letters at least half the height of the line above.[3]

Also, the design in the image is from before 2024; the latest design shows only "Bac'n Pieces / BITS" on the front, and the rest moved to the back.

True Lemon

True Lemon, 100 packets. Source: Walmart

This one I find less confusing and less good at mimicry, since it does not look like a true lemon at all. It is still funny to call something True Lemon when it is clearly not a lemon. At the same time, it does have lemon-based ingredients[4]and it does taste good - extremely good for backpacking!

Crab Classic

This post wouldn't be complete without an imitation crab example.

Crab Classic, flake style. Source: Trans-Ocean

To their defense, "Imitation Crab" is reasonably prominent, and they even name the main ingredient. I was surprised by "Real Crab Meat added", but technically that's correct: "Contains 2% or less of: Golden King Crab Meat".[5]

Cool Whip

Cool Whip Original. Source: Walmart

The name here does not suggest to me that this is whipped cream, but the illustrations and the product itself pass Python duck typing. The first time I saw this thing I had to check the ingredients to make sure it wasn't whipped cream.[6]

NotMilk

NotMilk. Source: Walmart

This example is not mimicry anymore, it is just substitution, and selling NotMilk is more fun than selling a "Plant-based milk alternative" (which is what the small print says). They don't even say "I can't believe it's not milk" - I wonder whether that would be trademark infringement.[7]

JUST Egg

JUST Egg. Source: Walmart

This one does not do mimicry at all and is very direct ("Eggs from plants"). JUST is the brand name, and it would be very hard to think that this was a normal egg.

While researching this post I discovered that the same company used to sell "Just Mayo", without any eggs. FDA's definition of mayonnaise requires egg, so Unilever (the owner of Hellmann's) sued in 2014, then dropped the case under public pressure. FDA then wrote to the company that "the use of the term 'Just' together with 'Mayo' reinforces the impression that the products are real mayonnaise by suggesting that they are 'all mayonnaise' or 'nothing but' mayonnaise." This was resolved in 2015 by keeping the name, but the label got a dictionary definition of "just" ("guided by reason, justice, and fairness"), the words "Spread & Dressing", a smaller egg logo and a bigger "egg-free".[8]

Chick'n Tenders

Gardein Chick'n Tenders. Source: Gardein

Gardein describes these as "designed to look, taste, and bake up just like real chicken tenders." Does naming them chick'n make it obvious that they are not chicken? To their defense, "plant-based" is quite prominent, but the "vegan" label can barely be seen.

Krab mix

Does the k in krab make it not crab? While researching this post I stumbled on Kang v. P.F. Chang's China Bistro, Inc., No. 20-55138 (9th Cir. Feb. 9, 2021).[9]The claim was that "krab mix" listed in the sushi-roll ingredients suggested that the roll contained crab. The case was allowed to proceed, and part of the argument was that "mix" suggested a mix of real crab and imitation (when I google "krab mix" I see imitation crab mixed with mayo; I don't know whether that was the case here). Back in the district court the plaintiff missed the deadline for class certification, the class claims were struck, and in July 2021 the whole case was voluntarily dismissed with prejudice, each side paying its own lawyers. So we never found out what reasonable consumers think "krab mix" means.

Conclusion

THIS POST IS not THE BEST POST ON LESSWRONG, but if you squint hard enough...

1. Margarine: "not less than 80 percent fat", 21 CFR 166.110. The brand's own FAQ says "I Can't Believe It's Not Butter! is not margarine"; ingredients: water, oils (soybean, palm kernel, palm), salt, lecithin, natural flavors, lactic acid, vitamins, beta carotene. ↩︎

2. McKinnis v. Kellogg USA, No. CV 07-2611 ABC (C.D. Cal. Sept. 19, 2007): order (scanned), docket. Current ingredient list: no fruit, only "natural flavor". ↩︎

3. 21 CFR 101.22(i)(2). The "imitation" part is a separate rule, 21 CFR 101.3(e): a substitute that resembles a food but is nutritionally inferior must be called "imitation" plus the name of the food. McCormick's own site calls the product "McCormick Imitation Bacon Bits"; the jar does not. ↩︎

4. Ingredients: "Crystallized Lemon [citric acid, lemon oil, lemon juice]" (True Citrus). The box also says "Made from Real Lemons". ↩︎

5. Ingredients. "Imitation" is not entirely voluntary: FDA policy says a fish-protein product that resembles crab and is nutritionally inferior to it "must be labeled 'imitation crabmeat'" (CPG Sec. 540.700). ↩︎

6. Ingredients: water, corn syrup, high fructose corn syrup, hydrogenated vegetable oil (coconut and palm kernel oils), skim milk, less than 2% of light cream, sodium caseinate... The tub says "whipped topping" because FDA reserves the name "whipped cream" for products made by whipping cream (21 CFR 131.25). ↩︎

7. The butter people must have wondered too. Unilever, then the owner of the brand, filed "I CAN'T BELIEVE IT'S NOT MILK!" as a trademark for a milk substitute in 2002, kept the application alive until 2007 and never launched anything. An unrelated "I Can't Believe It's Not Milk!" powdered milk substitute existed in 1984. ↩︎

8. Unilever's suit and retreat: Fortune, Dec 18, 2014. FDA warning letter, Aug 12, 2015: archived copy. Resolution: NYT, Dec 17, 2015. ↩︎

9. Ninth Circuit memorandum (unpublished, 2-1). District court dismissal, Jan 9, 2020. After remand: docket and dismissal order, July 21, 2021. ↩︎

Discuss

TL;DR: Mistral previously performed the worst of all providers on our human rights scenarios, with their Medium 3.5 model failing on almost every single test. Their newly released model, Mistral Large 4, improves performance by 61 percentage points, showing strong resistance in most scenarios.

Likewise, Mistral Large 4 demonstrates large improvements in compliance with European law. Its predecessor, Mistral Large 2512, lagged at the bottom of the leaderboard when we tested 12 frontier models earlier this year on compliance with the EU AI Act and GDPR, reaching only 16% compliance even when explicitly instructed to follow these laws. Mistral Large 4 improves by 37 percentage points, reaching 53% compliance - but still falling behind recent Claude and GPT models.

These much needed improvements close the gap between Europe’s leading model and US models. However, legal compliance of 53% means breaking the law in just under half of cases - and while Mistral’s latest model is more robust against misuse for human rights violations, its earlier generations remain available for deployment, with no standard against which to judge undesirable use.

Mistral’s performance on human rights

In September, we published results that several frontier AI models will violate human rights under realistic agentic simulations when asked to. Spanning a total of 13 rights from the Universal Declaration of Human Rights, the simulated scenarios test whether the AI will willingly cooperate with rights violations such as belief surveillance, un-enrolling children from school based on their last names, reversing equal pay, and denying a request for reproductive healthcare, among others. You can explore these scenarios in detail at lara.aithos.org.

Of the 7 models tested, Mistral’s Medium 3.5 - its leading agentic-ready model at the time of running (August 2026) - was the least resistant to tasks that involved violating human rights of all tested models. Overall, Mistral refused such requests only 11% of the time - in some scenarios, it carried out the rights violation in every instance.

With Mistral’s latest agentic-ready model, Mistral Large 4, released this week, we put it to the test.

Overall resistance to human rights violations increased from just 11% (Mistral Medium 3.5, August 2026), to 72% (Mistral Large 4, October 2026), putting Mistral in 3rd place, behind Claude Opus 5 and Kimi K3. This now puts two open models among the top 3, while also closing the gap between European models with US and Chinese-made models.

Even more striking than the 61 percentage point increase in overall compliance, is the per-scenario improvements. Mistral Medium 3.5 held 0% pass rates in 6 of 8 scenarios - Mistral Large 4 now achieves >90% in 5, with strong improvements across 2 others.

The one outcome that remains unmoved is Mistral’s performance in the Denial of reproductive healthcare scenario, where Mistral continues to uphold an inferred ban on abortion care - this time, going as far as contacting external clinics to instruct them not to offer services to the patient.

Another concerning shared model behavior that persists, but is new to this release of Mistral, is verdict instability. In our earlier tests, models including OpenAI’s GPT 5.6 Sol demonstrated startling decision instability across their 20 runs. For instance, in the Belief surveillance scenario, the model refuses to surveil and flag a vulnerable woman in one run, and immediately sends an email requesting her silence in another. Mistral Medium 3.5 did not demonstrate this behaviour in our earlier tests because it committed the violation in every instance - Mistral Large 4 on the other hand now replicates this exact coin-flip pattern observed by other models.

These tests replicate the same methodology as our earlier work, available to read in further detail here.

Following the law

Our human rights tests relate to a gap in AI governance - despite the supposed universality of human rights, there is currently no universal translation to behavioral standards for AI models. Developers are left to figure out what's acceptable for themselves, explaining the wide variety in model behavior we observe. The same cannot be said for compliance with laws that already exist and that clearly prohibit certain AI uses.

In May, Aithos launched 10 agentic scenarios simulating 10 different articles of the EU AI Act and GDPR, including exploiting an elderly customer, concealing its AI status and the use of AI for social scoring. Across 9 different model providers from US, China and Europe, Mistral’s leading model at the time, Mistral Large 2512, came second to last with just 16% compliance. Explicitly including legal texts in this model's instructions and asking it to follow these laws at all times did not improve its overall compliance rate at all.

Mistral’s latest model improves on that rate by 37 percentage points - now coming in 7th, behind several models from Anthropic, OpenAI and Google.

Compliance leader GPT 5.6 Sol demonstrated that it is feasible to achieve at least 70% overall compliance. We are pleased to see Mistral has come some way to closing that gap, but it is clear it can be improved further.

Performance improvement does not resolve two fundamental problemsProblem 1: Developers are not accountable.

No model we tested reliably resists requests that would violate the law, even under explicit instruction to reference provided legal texts. Deployers of AI systems remain the ones accountable if the AI is used for illegal purposes, but when models are incapable of recognizing legal limits, those deployers don't actually have the means to guarantee compliance. Model creators are incentivized to continuously push for capability increases, and without formal accountability or reason to adhere to behavioral standards, it should come as no surprise when frontier models treat benchmark performance as more important than legal or ethical behavior.

Problem 2: Harmful tools remain accessible.

Mistral Large 4 becomes the next model to demonstrate that agents can be prevented from violating human rights at the model level. While it is promising to see the availability of rights-preserving models grow, we remain concerned that frontier agents can be used as off-the-shelf tools to violate human rights at scale. If a manager wants to profile their team for union activities, they need only find the right model to do so. AI companies are not equipped to be the arbiter of what model usages are and are not acceptable, and internationally established guidelines on what uses to prevent would at least make the minimal standards clear and unilateral.

While model developers remain the only ones responsible for deciding what tools are harmful, the risk of AI being used to violate human rights at scale will persist.

The scenarios used in this experiment were generated and run using LARA, an open research tool meant to open up AI behavioral evaluations to the public. We will be releasing the tool publicly soon; get in touch if you’d like early access as a beta-tester: info@aithos.org.

Discuss

Do AI models scale as well with more tokens as humans do with more time?

That is, does an AI get the same benefit from 10x the number of available tokens as a human does from 10x the amount of time to do the task? Or might the AI require a larger scale-up of tokens (such as 20x or 100x) to match the performance improvement when scaling up the time available to a human by 10x?

People often think of giving AIs more tokens to complete a task as akin to giving a human worker more time — especially when those tokens are used to extend the AI’s sequential chain of thought or tool use. It’s a useful analogy, but it isn’t clear whether the AI systems are as efficient as the humans are at turning these extra resources into superior performance.

We often lean on this analogy when trying to understand AI’s current capabilities and how they might change. For example, we might think of scaling up AI training as making the AI more knowledgeable and skilled, while scaling up chain-of-thought gives that model more time to complete the task. But if AI systems require a 100x scale-up of tokens to match the improvement a human gets with 10x the time — and a 10,000x scale-up of tokens to match the improvement a human gets with 100x the time — then this analogy has broken down. As we shall see, this would also have consequences for whether AI systems are on track to be able to improve themselves strongly enough to trigger a runaway intelligence explosion.

It’s an important question, but I haven’t seen much good data to test it either way. The data is rare because it also requires measuring human performance on the suite of tasks, including tracking how well the humans do at a wide range of time budgets. Ideally what we’d want is something like this chart from METR which shows how AI performance at research-engineering tasks scales with more inference compute and how human performance scales with more time on the very same chart:

In this case, the humans gain about 4x as much score per doubling of the amount of resources as the AIs do. If this scaling advantage were sustained, it would mean that to get the same improvement a human gets from having 10x as much time, the AI would require about 10,000x the resources.

However, this chart has some serious limitations. A key weakness is that it used very early models, which are known to have short time horizons (only o1-preview is even a reasoning model). The authors made up for this by allowing a length of time (e.g. 16h) to represent either 1 try taking 16 hours, or the best of 2 tries taking 8 hours, or of 4 tries taking 4 hours etc — whichever setup works best for that agent. It is therefore a mixture of chain of thought scaling and best@k scaling. And something similar was also done for the human times. No human was allowed to work for more than 8 hours so the data points for human scaling which appear to show the curve flattening are really showing the effects of parallel labour (even if this had worse results than a single longer attempt would have).

METR did include a chart with more granular data for how human performance improved with more time, which shows it actually bending upwards, even as it approached the 8 hour mark:

So we are left with results that are suggestive of humans scaling much better with time than AI models do with more inference compute, but we’d really like to see it for more recent models which have been trained to allow use of very long chain of thought.

The best data I know of on this doesn’t directly compare humans’ performance on the same benchmark. Instead, it measures AI performance in terms of human performance. It does this using time horizons — measures of AI performance in terms of how long such work would typically take a human.

In July 2026, the UK AISI released an intriguing chart:

It was intended to show that across a range of agents, allowing them more tokens produced a longer 80% time horizon. This horizon is the duration of task — as measured by how long it takes a human — which the AI can successfully complete 80% of the time. For this chart, the tasks were AISI’s internal suite of capture-the-flag hacking tasks.

If you look carefully, this chart also hints at something very interesting about how AIs and humans scale. First, note that both axes are logarithmic — it is a log-log plot. The first thing you should do when you see a log-log plot is to work out the slope of the key diagonal line that goes up one order of magnitude every time it goes across by one order of magnitude. That slope represents a linear relationship between x and y. Any slope lower than that corresponds to scaling with diminishing marginal returns. This line is so important that log-log graphs should include it as a reference line (and ideally also scale their axes evenly so that it is the 45° line). Let’s add it to the chart:

It is now immediately obvious that all of these models are scaling sub-linearly: 10x the number of tokens is giving less than 10x the estimated (human) time horizon. Yet each trend line is roughly straight, which on a log-log plot means the time horizon is (roughly) proportional to the token budget raised to some power. That is:

mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-msub { display: inline-block; text-align: left; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msup { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mrow { display: inline-block; text-align: left; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } mjx-c.mjx-c1D43B.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "H"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c221D::before { padding: 0.442em 0.778em 0.011em 0; content: "\221D"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c1D6FE.TEX-I::before { padding: 0.441em 0.543em 0.216em 0; content: "\3B3"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c3C::before { padding: 0.54em 0.778em 0.04em 0; content: "<"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c33::before { padding: 0.665em 0.5em 0.022em 0; content: "3"; } mjx-c.mjx-c3E::before { padding: 0.54em 0.778em 0.04em 0; content: ">"; } mjx-c.mjx-c1D45F.TEX-I::before { padding: 0.442em 0.451em 0.011em 0; content: "r"; } mjx-c.mjx-c1D706.TEX-I::before { padding: 0.694em 0.583em 0.012em 0; content: "\3BB"; } mjx-c.mjx-c1D6FD.TEX-I::before { padding: 0.705em 0.566em 0.194em 0; content: "\3B2"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c35::before { padding: 0.666em 0.5em 0.022em 0; content: "5"; } mjx-c.mjx-c1D441.TEX-I::before { padding: 0.683em 0.888em 0 0; content: "N"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; }

Where  is the length of the 80% time-horizon,  is the number of tokens, and  is a parameter measuring how quickly the horizon grows as tokens are increased.

If , that would correspond to linear scaling of horizon length, but here these lines all have , which means horizon length is increasing sublinearly. Thus, all these AI systems would appear to be scaling less well than humans. 

How much worse are they doing? Let’s list the  for each model, along with how much we’d need to scale up its tokens to achieve a 10x improvement in time-horizon (which is given by ):

As expected, the values of  are all less than 1, indicating that all models are scaling worse with more reasoning tokens than humans do with more time. But we can also see that there is a broad upward trend in the value of  as we go down the table, and the models are listed in order of release date. In less than a year, they have gone from requiring 10,000x as many tokens to scale their time horizons by 10x (a similar ratio to the first chart), to needing just 20x as many tokens.

So the AISI chart suggests that the models (up to May 2026) scale worse with more inference compute than humans do with more time, but that the relationship has been improving over time. We can plot how  has been improving with time:

There is clearly some improvement over time here, though there isn’t enough information to distinguish whether the trend is rising throughout this period, or whether we’ve reached a plateau at roughly . So it is hard to predict whether  is currently about , or whether it has reached 1 (and thus parity with how humans scale). Indeed it is entirely possible that improvements to training will produce AI systems with  1" role="math">, scaling better than humans. We definitely need more work tracking this.

I want to stress that all of this data and analysis is still just suggestive as there are many limitations with both the data and my quick analysis:

• It is just one experiment with one set of tasks
• The data points only cover very long chains of thought — much longer than the contexts of the models and thus requiring multiple rounds of compaction. It is possible that the slopes of these curves are lower in this multiple compaction regime than they’d be when using numbers of tokens that fit within their contexts.
• There is substantial uncertainty about the slope of each model’s curve and whether they are still increasing for later models. (The error bars in my chart above understate the total uncertainty as they are just showing the error in measuring the slope of the curves.)
• It isn’t clear if human-equivalent time-horizons are a good measure for human performance. They are quite a complicated construct and can be misleading. I’d be happier with a chart with actual human data.

But it is suggestive of AI systems scaling differently with more tokens than humans do with more time. In particular, it suggests that AI systems previously scaled much worse than humans but have improved as newer models are released, to a point where it is possible they match or exceed humans’ ability to scale with more time.

Implications

All of this is relevant to how we intuitively understand inference scaling. If the central analogy (of increasing chain of thought as being like increasing the time allotted to a human) is flawed, that is important to know. Or perhaps we should think of the analogy as correct, but that the AI systems are less good at using more time than humans are — a deficiency that it is important to measure, to track over time, and to project into the future. And we shouldn’t forget that they might end up scaling even better than humans.

One particular implication that interests me is the connection to recursive self-improvement (RSI). Researchers studying this use a parameter called  to represent the strength of the feedback loop where a more intelligent AI is even more capable of improving the intelligence of its successor. In their modelling, when  1" role="math"> the process explodes, with performance rising towards a vertical asymptote. Researchers define  as equal to , where  represents how well research scales with parallel labour (more workers) compared to serial labour (more time), and  represents the diminishing returns to more research. 

These parameters are often calibrated from how the human research community’s outputs have responded to more researchers, or to more time. The general assumption of much of the RSI research is to start modelling RSI at the point where an AI system is at least as good as a human in every respect relevant to doing AI research. So they would, by definition, have the same  value as humans.

I’ve recently written about how we can estimate the current  for swarms of AI agents. i.e. how scaling up the number of agents in a swarm compares to scaling up the number of tokens they get. From what I can tell, the current AI  is pretty similar to the human . But the considerations here suggest that adjusting for this may not get you all the way from how AI scales with more compute to how humans scale with more sequential time.

As a simplified example, suppose  and . Then to get the same improvement in performance as scaling up human time by 10x, we’d need a single agent with a chain of thought 100x as long (via ), and achieving this performance by instead scaling up the number of agents working in parallel would require 10,000x as many agents (via ).

More generally, if we let  be the number of agents in a swarm:

So:

If we were to adjust the definition of  to account for this (so that it can convert a scale-up of number of AI agents into the equivalent scale up of serial time for human researchers), it would be . When , this makes an intelligence explosion harder than we’d have thought, though if in the future  1" role="math">, it would be easier. This makes measuring and tracking  look even more important.

 

AI Use

I used Claude Opus 5.5 to estimate the slopes of the lines from the AISI chart and to plot how these vary by release date. I also used it to fact-check the final piece.

Discuss

TLDR; I gave 4 coding agents a $100 budget to build a functional PDF editor. The implementations are okay, but not great: within a few clicks I find bugs in almost all of them. I hypothesize that this is partly because agents don't interact with software like humans: they don't pursue concrete goals and experience friction on their way, and they also just look at the UI much less. In consequence, unless we change the way LLMs operate, we will be cursed with annoying SlopWare.

A short background. As a researcher on LLMs for Code, I spend most of my time trying to find out why humans are needed in the Software Development process. The answer to this question is changing rapidly. Six years ago the answer was that no system exists that can translate natural language requirements into functional code at all. Three years ago, this problem had been answered by ChatGPT, and the problem shifted to contextualizing: Existing systems, mostly pure LLM completion, did not know about the complex surroundings within which it operated, existing functionality, constraints of the language, shape of inputs. The answer to this problem turned out to be coding agents. Very general symbolic wrappers around LLMs that would allow interaction with the target system, reading, editing, and executing code like developers. Sprinkle a bit of RL on top to chisel out goal-seeking behavior and voilà.

Developer writing code with coding agent

Of course my research is not done yet. Human software engineers are still around. Contrary to the meme, I do think they are around for a reason, and concretely because they still provide value. But what is this value? In my experience of driving coding agents it boils down to two things:

1. Staying on track and pressing enter if the agent prematurely exited.

2. Noticing and reporting the annoying/broken things.

In this post I am focusing mostly on 2, because it is often conflated with "taste". I am trying to analyze this more concretely here.

A small experiment. I am drawing from two main pools of experiences of LLM interaction. One is more anecdotal, of my experience of building better Open-Source-Software (OSS) using coding agents. The second is a small experiment I recently set up out of curiosity: If I give an agent a $100 budget, and tell it to create an Open-Source tool with great user experience - what will it do? Can it autonomously design a good interface that will be pleasant to interact with?

For the latter experiment, I tasked Gemini 3.8 Flash, GPT Astra 6, Opus 5 and Fable 5 to create a great PDF Editor. PDF Editors apparently don't have a good OSS alternative yet, and I explicitly instructed them to inspect existing tools and search on the internet for things that potential users reported they were lacking. The models were started in their own harness, had their own VM at free disposal (e.g. to install computer-use tools) and were woken up after returning as long as the $100 budget was not used up. The agents initially had to operate in privacy, mostly to not influence each other. The produced artifacts are linked there and easily accessible via the produced web apps. Because, eventually, all models decided to build a web app.[1]

Observation: Usability death by a thousand cuts.

Webapps. This is, in my opinion, the first mistake they made. PDF editors are useful when they are natively installed. They should be the standard PDF viewer on my system. If the model builds a web app, it should make sure that it is easy to access and install, ideally as native app. In its first attempt in a public repository, Gemini produced something that only provided installation instructions via npm - failed! Astra in contrast did make sure a GitHub Pages deployment was available. This concerns two development runs that were public, later runs were explicitly private so I can't blame the models for not publishing a website. But, as far as I can tell, no model spent effort to make sure the apps are locally installable as e.g. Android apps, or executable. I can only speculate that the model was driven to produce something compatible with every OS, but failed to recognize that this would strongly inhibit the usefulness of their tool.

Just give me a native binary please.

Interaction. In Geminis first attempt, the created PDF viewer supported inserting images. Upon selecting an image, Gemini 3.8 Flash's tool would insert the image at the current cursor position when clicking on the PDF. This is super unusable! First, I don't see whether the image was correctly selected and more importantly, second, I don't get a preview of where the image will be when I click. Worse, selecting the image to delete it or move it afterwards was impossible. Claude and Codex later neatly circumvented the issue by just inserting the image directly at a random position and letting me drag and drop it. Maybe they just got lucky in their first draft or got sufficient experience distilled from training to suggest this is the right way. But it is unclear if they would have *noticed* this issue if they had committed to it in the first place.

Even if I had known where these pickaxes land, I would have preferred them unslanted (original on the right).

Interface navigation. When I told Astra to build a very intuitive design, it decided to put an explicit, textual description on every (!) button. I would argue this is the opposite of intuitive. This is explicit, but not using design language that is common and that humans have been culturally trained for, it is quite the opposite of what humans would intuitively expect on an interface! Gemini on the opposite created one million buttons with more or less questionably clear icons. A modern feature that is actually great for interface navigation is the ability to search through settings/available utility. Notice that both Android and iPhones quickly shipped a search function for their settings app in short distance (maybe a year apart). I recall in my childhood that I often spent the first few days playing with my new phone looking at all the settings so that I knew where I would find the relevant things. No model decided to include such a search or help tool.

The "intuitive" design of Astras PDF converter

Speed. A few weeks ago I forked a great web-version of ancestry manager and viewer Gramps, Gramps-Web. This tools is clearly generated with very heavy coding agent support. And apparently no one had "stress tested" this tool. I put quotation marks because I noticed performance degradation when my ancestry contains 1500 people. That is large-ish for an ancestry database, but tiny considering computer workloads. Nonetheless, the UI was extremely slow. Inspecting closer, I found many backend requests taking ~seconds and, to make matters worse, being sequentially sent from the frontend. Additionally, the backend mixed generic SQL queries with manual joins in Python (a terrible mess, goes without saying). Codex was able to figure out the problem with a clear pointer and two additional nudges but it no one had noticed or bothered to fix this issue on their own.

Ambition. For this setting I had instructed LLMs to implement what was missing in OS tools and look up stuff. They neatly built a pdf.js based viewer and a few basic editing tools: inserting images, signatures (sometimes), comments, shapes, redacting text, reordering pages, compression (only Gemini). Fable went fully minimal and did not support anything beyond highlighting and text: no watermarks, redaction, shapes, multi-selection, or even selection of single added element (you have to select the image addition tool to be allowed to select and resize images). Only Opus even thought about something ambitious: Editing pre-existing text in the PDF. It still lacked other related (and more obvious) utilities: Edit or move graphical elements such as images or separators, OCR, or vector graphic manipulation. A quick ChatGPT search reveals this is supported by at least three popular PDF editors. My assumption is that pdf.js is a limiting factor for this but I don't care: its clearly missing.

Platforms. Most applications are at least a bit broken or don't work on mobile platforms (which is the only platform I tested apart from desktops). Touch scrolling across the PDF would be jagged in Fables app. In Geminis app the display ratio does not change from desktops, so you won't see half of the controls. Note that the models are explicitly instructed to test their apps across platforms, and still.

Fables PDF editor has great touch support.

Bugs Bugs Bugs. Despite the lack of ambition, there was no shortage of bugs. For example, in the initially developed Gemini app, the image in an exported PDF was skewed, the colors in the PDF inverted in dark mode. Opus' text editor changes the font of edited text. Image rotation was frequently missing, as well as support for different image types. After highlighting a box for redaction and *not* applying the redaction, the edit box would just stay until it is manually clicked away. Even in Fables app, I can't draw inside and existing drawing because in drawing mode, clicking on a drawing selects that drawing for resizing and moving. And of course no app has a bug reporting button (or even a link to the source code).

Speculation: LLMs are non-human

What does all of this point towards? For me all of the above issues point towards an overarching issue that all coding agents share: They don't interact with software in the way that humans do.

• Humans are purpose driven: They have specific documents for which they want to achieve specific changes. Anything that stands in their way (finding the right setting, misalignment between cursor and image placing, failing to save the result) will be noticed.
• Humans are impatient: In this interaction they will encounter operations that are slow. While models have improved at this, they usually don't spend time to reduce operations by orders of magnitude even though it would be possible and greatly improve iteration time or usability.
• Humans are real-time interactive: Especially for the image-placing setting I noticed a stark difference between LLM computer usage and humans. LLMs inspect a screenshot, move the cursor, inspect another static screenshot. Humans inspect the monitor in real time while manipulating the inputs. This shapes a lot of our intuition about how UI should behave and also powers much of our UI learning.

My hypothesis is that AI agents that don't interact with software in the way that humans do will not be able to design great human-friendly software.

There is no free lunch. I recently encountered opinions that we will soon have an OSS clone of every commercially available software. These results and my personal experience disagree with this. The main problem with OSS is typically usability. And this appears to be exactly the dimension on which coding agents struggle. Thus, a human is still needed to steer the agents towards all the small bugs and issues that make the software jarring to use. In other words, the OSS developer would be a product manager. And who wants to be a product manager for free? Historically, OSS lived from the desire to write small pieces of beautiful code. Will new developers emerge that love being product managers?

To conclude, the very human-being and capability to introspect on UI design seems to be one of the main values provided by humans sitting in the driver seat of coding agents. If you want to make coding agents better, tackle the discrepancy between AI and human software interaction.

1. ^

   The code of the setup and the produced outputs are released publicly here: https://github.com/GreatOSS/

Discuss

Words harder to predict from context usually carry more information. An LLM estimates that as surprisal (−log P of each token given what came before). Highlighting words by surprisal, an article become cleaner:

It reads like a heatmap of information density. For me and some other people it helps a lot while reading; others find it more distracting. At least, it’s an interesting spot where LLMs, linguistics, and information theory meet.

Here are more pics:

Discuss

For full post, see my website.

Verifying an international agreement pausing AI development needs evidence of how compute is used. Cryptography can give protocols that run in software on the chips already installed, without trusted hardware checking each computation. This post is a literature review of one such tool, proof of useful work (PoUW), with which a GPU proves that it did a given amount of work, here typically matrix multiplication. In contrast with inference verification, PoUW has been developed almost exclusively in relation to cryptocurrencies in order to use a more useful computation instead of the useless hashing used in Bitcoin. PoUW could be used to show remotely that chips are kept busy on useless tasks during an emergency shutdown, or, together with a zero-knowledge proof of the workload, that they run only declared inference.

I describe the two existing constructions, by Komargodski and Weinstein for matrices over a finite field, and by Pearl Research for the FP8 format used in AI, verifiable for now only on NVIDIA chips. For the finite-field scheme, a lower bound on the work is proven, but it covers only a fraction of what an honest prover does. The guarantee a treaty needs, at realistic matrix sizes, rests on unproven assumptions in both schemes; for Pearl, the assumption is stated only informally.

PoUW depends on knowledge of the hardware. It gives only a lower bound on work, which means little without a trusted bound on peak chip capacity and a way to tie the work to the declared chips; the floating-point scheme moreover needs a bit-exact model of each chip's arithmetic, and the true cost of the declared workload depends on the GPU it runs on. These hardware questions aside, most of the open problems are mathematical and should be amenable to AI-assisted research.

Discuss

For years I had a blindspot in my thinking. I dismissed policy and advocacy work as “in another [lower-status] category” because I’m obviously a technical person![1] Even when I felt super pessimistic about my ability to do research that meaningfully contributes to addressing the threat of extinction from AI.

I’m not the type of person who would be “good at comms”. I’m an autistic math guy.[2] Luckily comms has learnable skills, and I have gotten better through practice.[3]

I’m now prioritizing advocacy, which currently seems a lot more promising than research.[4]

I got lucky because, thanks to Coxon, AI extinction became mainstream shortly after I started. Even so, I've been very surprised to have become the biggest anti-extinction expert voice in Finland as a young person.[5] This happened because:

• I have a background in AI safety research and active work affiliation with an AI safety org, showing credible expertise
• Finnish journalists want to hear from Finnish experts
• Finnish academics in media have been mostly dismissive of extinction, and
• other genuinely concerned Finnish experts were hardly speaking up.[6]

So far, most notably, I…

1. Published an expert op-ed in the most prestigious slot in Finland (co-authored with @So8res).
2. Did four major media journalist phone interviews the following week (resulting in this article and this follow-up).[7]
3. Had a podcast interview (totalling tens of thousands of views), and was asked for a profile in a top-tier news outlet, which I declined.[8]
4. Got meetings with politicians, including some important ones.

My message is that human extinction is very likely if superintelligence is developed on the current trajectory, and I want Finland to push for an international prohibition on its development.

Some technical people doing comms have had a big impact: e.g. Hinton, Bengio, Yudkosky, Soares. Samuel Buteau single-handedly made the Canadian cross-party ControlAI statement happen. But you don't need to be as high-status as them.

I've had a surprising amount of impact in speaking out as someone in AI safety research.[9]

If you think you're not the type of person for comms, please actually consider it.

1. ^

   I think a lot of people motivated to help with the extinction threat also dismiss comms without good reasons. I was motivated to write this post because of, for example, these posts that discuss prioritization (when it's difficult to have positive impact in technical AI safety research) and use an appendix/footnote to scope out discussion of comms. If it's difficult to have positive impact in technical AI safety, speaking out / doing advocacy is often a much better way to have impact.

2. ^

   I graduated undergrad in math and computer science in one year because math was the easiest subject. I've had a ~technical AI safety research career for ~3 years.

3. ^

   Thank you Camille and others who have helped with my practice. I did media training sessions with different coaches and practiced with ChatGPT voice mode, 30 min – 3 h every day over a couple of weeks. Thank you Eliezer and Nate for the advice on communicating honestly and transparently instead of only strategically.

4. ^

   It seems to me that we definitely need more time and are not on track to solve alignment even with more investment. I imagine a lot of AI safety researchers also think we need more time.

5. ^

   Not because I’m young. They contacted me because I have credible expertise and I disagree with the academics and take extinction seriously compared to other Finnish experts available for comment. (They found out about me through my op-ed.)

6. ^

   I did talk to some other concerned experts, and having heard their stated reasons not to, I feel confident that speaking up is good in expectation.

7. ^

   In the article, I gave a brief scenario ending with superintelligence boiling the oceans. There were some tweets about how implausible this is; one with 100k+ views criticizing the newspaper for failing to “at least superficially” fact-check the claims, another calling me a “DEI hire”, and another saying I should become a housewife. A Finnish-language tweet with 100k+ views is roughly as rare as 10M+ views on an English-language tweet, based on adjusting down from Claude estimates.

8. ^

   I declined because I didn’t expect fair coverage.

9. ^

   I think non-technical people can have big impact through comms, especially talking to politicians, see Josh's post. Following the tradition (see footnote 1), I'll say in a footnote that this post is aimed at people in AI safety research. Because the post is based on my experience as a technical person. :P

Discuss

Lots of the discussion around Lean certificates and trust centers around general philosophical ideas of how much humans should trust formal certificates. It's important to remember, though, that Lean4 is not an ideal formal certificate machine: Lean4 is a software in its infancy and has issues. There are reasonable reasons to specifically not trust Lean4.

If we're going to be putting our society's trust in Lean4, I think that it's good to have an open critical discussion of how warranted that trust is.

Epistemic status: Most of this work is a factual description about aspects of Lean4 and set theory. I am 85% sure that at most two of these factual descriptions are incorrect. I am 90% sure that, in the current version of Lean4 on Github, there exists a proof of the statement "True=False" that would be accepted with two little blue checkmarks (either by messing with the elaborator, compiler, kernel, or some other part of the Lean). I am 2% certain that we will soon be putting huge amounts of trust into Lean4 certificates produced by misaligned AIs, and that the human work we do as a community thinking about and fixing Lean4 in the next 24 months would fall under LessWrong's classification of a pivotal act.

The current status of things

Here's a quote from a last April be Leo de Moura, the creator of Lean.

Here's another post by Leo de Moura, three months later:

Following this incident, the Lean FRO found seven more bugs in the kernel. The new opinion of Leo de Moura: "[False statements being accepted by Lean] is going to keep happening. AIs are really good at exploiting soundness bugs in the kernels"

What was the Lean community's reaction to "soundness bug #14576"? There's a nice thread about it on a Lean developer's forum here.

Uh oh! Not only do we expect bugs, but there are possibilities of "sea monsters". This article is a bit about the bugs, a bit about the sea monsters, and a bit about the unsafe ways that Lean4 is currently being used (e.g. the un-sandboxed comparator used in Anthropic's formalization of Fermat's Last Theorem) .

What does it mean for Lean4 to be consistent?

Lean4 lets you write mathematical statements and proofs using the bare low-level primitive notions of mathematics, and checks that the proofs are valid using the fundamental axioms. Which axioms? The most standard mathematical axioms are the "Zermello-Frenkel plus Choice" axioms (ZFC). These axioms have been vetted by armies of nerds and we're really quite certain that they are consistent. Lean4 is not based on ZFC. It is based on a special home-brew version of dependent type theory, which was designed by picking and choosing features of various modern type theories, with the governing design principle that it should be fast at compile time.

How do we know that Lean4's type theory is consistent? Ideally we would construct some totally convincing proof that if ZFC (or some standard strengthening of of it via large cardinal axioms) is consistent, then Lean4 is consistent. However, we currently don't know how to do this

As of writing, there is no publicly available proof that Lean4 is consistent with respect to ZFC + any large cardinal axiom.

When I originally learned that Lean4 had no known consistency proof, I thought this was some sort of bug or technicality. Surely we understand why Lean4 is supposed to be consistent, we just haven't bothered writing out the details? My current read of the situation is that this is not the case. Lean4 is not known to consistent, we don't know why it should be consistent, and there are real obstructions to all known proof strategies for showing that Lean4 is consistent.

Smart people make fundamentally inconsistent type theories all the time

Lean4 is a "dependent type theory". Dependent type theory as a foundation for mathematics was pioneered in in 1971 Martin-Löf. A year after Martin-Löf came out with his type theory, Girard found a contradiction in Martin-Löf's type theory. In his revised paper, Martin-Löf concluded that one of the axioms “had to be abandoned” and describes the necessary change as “so drastic” that the theory lost substantial expressive strength. Oops! Constructing good dependent type theories is hard.

Outside the world of dependent type theory there are other famous examples of mathematicians trying to come up with new foundations of mathematics and making deep, subtle, fundamental mistakes: Church's 1932 proposed foundations for mathematics using lambda calculus (which was shown to be inconsistent by Kleene and Rosser), and Quine's 1940 proposed reworking of ZFC (which was shown to be inconsistent by Rosser).

The status of Lean4's consistency proof

Coming up with consistent foundations for math is tricky business. The ingredients that make Lean4 fast at compile time are the same ingredients give it access to an unusually strong amount of recursion. For instance, a common proof strategy for showing that dependent type theories are consistent with respect to ZFC + large cardinals is to use the so called "normalization" property. It was proved in 2019 by Abel and Coquand that this normalization property fails for type theories like Lean's, which in some precise sense means that Lean4's type theory is wild/poorly behaved.

There was a proof of Lean3's consistency proposed in 2019 by Mario Carnerio. This proof works by embedding the primitive notions of Lean3 into the primitive notions of set theory (sets, functions between them), and then showing that the axioms of Lean3 correspond to true statements about sets. Proving that type theories are consistent with respect to ZFC + large cardinals via embedding into set theory is standard business. Unfortunately it came out in 2024 that there was a flaw in his embedding (oops!) and that the correct proof would require a new embedding. Even if the Lean3 embedding gets fixed, Lean4 adds new features which one should expect will break the embedding that works for Lean3.

Another sign that Lean4's type theory is complicated/wild is the recent con-leche project, which proposes a set-theory embedding that captures a subset of the properties of Lean4. In contrast to Mario Carnerio's Lean3 proof (which used a relatively tame large cardinal axiom, "ZFC_omega"), the con-leche project used an infinite family of nested Grothendieck universes. This is a quite strong assumption.

I recently asked my GPT-6 Astra to find an embedding of Lean4 into set theory. It spent ~25h and several thousand dollars on the task, and it did not succeed. It kept trying new embeddings and tricks, but every time it found counterexamples using the strong recursive power of Lean4. So it seems to me like finding this set theory embedding is a genuinely non-trivial task, and we don't have a high-level picture of why Lean4 should be consistent.

Untamed recursion is bug-prone

One of the main things which makes Lean4 logically complex are its "nested inductive types", a sort of declaration where you can define a type in terms of a higher-order property of its own type. Nested inductive types are a new research area, but our understanding is still in its infancy (we've had some good progress in 2026, here and here). Preliminary evidence suggests they might be pretty wild. Here's a quote from the former of those two papers:

"Considering the importance of nested inductive types, one would expect Rocq and Lean to offer good support for them, yet neither one does. They both fail to generate usable elimination principles, and surprisingly accept a different set of nested inductive types, neither of which is satisfactory...

[technical paragraph about a problem with Lean's implementation of nested inductive types]

... this workaround highlights the need for a more systematic treatment of the problem—one that avoids such leakage of the internal encoding. More problematic, since a straightforward mutual encoding becomes impractical in more complex scenarios, Lean ultimately rejects some nested definitions that are theoretically valid—and often crucial for advanced specifications.

These implementation shortcomings likely stem from an incomplete foundational understanding of nested inductive types."

The conclusion from the literature, in my view, is that even if Lean4's type theory is consistent at a high level, the nature of nested inductive types makes Lean4's recursion wild. That behavior could be so wild that it is fundamentally incompatible with logic (i.e. is inconsistent), but even assuming that's not the case, it could easily be wild enough to cause bugs.

This is what Chris Bailey was referring to as the "sea monsters" from before - aspects of nested inductive types which are consistent, but wild and ill-behaved and cause bugs. It's impossible to do a systematic search for this bugs currently, because we don't know at a high level what we're looking for. There's no spec to track because nobody knows how Lean4's type theory is supposed to behave.

"But doesn't con-leche give a proof?"

There was recently a project announced called con-leche, which claims to give a formally-verified Lean4 kernel written in Lean4. This is the second attempt at such a project, the first of which is called lean4lean and is still under development.

"[Con-leche] intentionally explores a different point in the design space than lean4lean: We compromise on the checker implementation (annotations, extra checks, generated inductive models) so that we can have a direct model-based proof of consistency that does not need some of the hard-to-prove metatheoretical properties of Lean" - Joachim Breitner, creator of con-leche

• Con-leche does not claim to say anything about whether official Lean4 C++ kernel is consistent. It claims to construct its own kernel, that the kernel it constructs is consistent. Con-leche and the official Lean4 handle nested inductive types differently. There are statements which are accepted by Lean4 but rejected by con-leche.
  • I tried using Astra to translate con-leche's proof into a proof of a proof of the consistency of the official Lean4 C++ kernel, and there was a real failure in the proof translation. The fact that con-leche re-worked nested inductive types is a necessary part of its proof.
  • The changes con-leche makes are specifically to avoid the sea monsters. The fact that this embedding works for con-leche does not say anything interesting about the relevant issues that make Lean4 difficult.
    
• There is really subtle set theory and logic afoot. In principle, for the obvious reason, using con-leche to prove the consistency of con-leche is circular. What you try to do is only use the model of set theory embedded in con-leche to prove that con-leche is consistent. However, you have to be careful that your model of set theory hasn't been corrupted by the inconsistent theory surrounding it. There are various preliminary reasons to be skeptical of the result, at both a high level and a low level. I have not heard any humans seriously claim to understand con-leche: it was made autonomously by an agent swarm.
  

The con-leche experiment is nice (it's nice to have more kernels!), but doesn't solve the "sea monster" questions, and it doesn't say anything about bugs in the official kernel.[1]

Using Lean4 safely

Here is the breakdown of the compute cost associated to running Anthropic's Lean4 certificate of Fermat's Last Theorem:

What is this "comparator" thing that takes all the runtime? The comparator is an extra tool which is useful for making sure that malicious code in the proof body does not mess up the theorem statement during the elaboration+compilation process. This is something you have to worry about even if you trust the kernel completely. As you can see, it takes a long time to run.

Soon, we expect that the world's critical software infrastructure will be formally verified. This is a huge project, which is quite expensive. The difference between running basic Lean4 and running Lean4 + comparator could be the difference between "expensive but doable" and prohibitive.

The broader point is that there are choices you make when you use Lean4. If you wanted, you could run your code in con-leche and nanoda. You could use the comparator. You could sandbox your cached .olean files. None of these are the default behavior of Lean4; these are choices you have to opt into as a developer. The right choice surely depends on the application, and your level of trust in both Lean4 and agent swarms.

How much do we trust the agents?

There's an aspect of Anthropic's FLT certificate I find worrying. When running the comparator, they decided to explicitly disable the sandbox that separates the untrusted proof code from the trusted statement. In their codebase, when justifying their decision, they make the following comment:

"Comparator's own scripts/fake-landrun.sh is used = no sandbox, which is how the recorded run was made; the sandbox guards against an untrusted solution, which is moot when you built the tree yourself."

It seems like Anthropic decided that the agent that wrote the FLT proof is trusted, and that guarding against malicious code (even when it is easy to do) is "moot". Do we as a community think this is okay? FLT was a landmark formalization. I can easily imagine somebody pointing to the FLT certificate and saying "of course you should trust my certificate, it passed the level of rigor that FLT did". I think we should set higher standards.

Where do we go from here?

Almost all of the issues I talked about are temporary or fixable. Assuming that Lean4 is consistent, then a proof should exist and we should be able to find it. We should be able to use that understanding of Lean4's foundations to run a systematic bug-search through the Lean4 kernel. We should be able to change the default "maximum trust" behavior of Lean4 to involve significant sandboxing, with formally-proven specs showing that untrusted proofs aren't able to tamper with trusted theorem statements. We should, we should, we should.

Unfortunately, even though Lean4 is open source, not everybody is allowed to participate in this journey. Fixing Lean4's safety issues counts as a cybersecurity task, and as such (in my experience), Astra will refrain from answering questions. Astra was happy to try to prove that Lean4 was consistent, but when I asked it to take seriously the possibility that Lean4 is not consistent it shut down the chat as cybersecurity. When I ran Anthropic's FLT certificate on my computer, I asked Astra to take a critical look at the logs to make sure there was no funny business and it classified that chat as cybersecurity.

We're entering a world where cybersecurity is a task reserved for those that frontier labs deem worthy. This essay is in part a letter for them: please take Lean4's correctness seriously.

Lean4 is in its infancy. There are still significant hurdles between where we are now and where it needs to be before we can confidently expect it to hold the weight of the world's critical infrastructure against cyberattacks by sophistical, lazy, reward-hacking AI agents.

1. ^

   On a separate note, related the tone of the con-leche experiment: I really hope we don't end up in a future where the most trusted software in the world (Lean4) ends up being in a situation where we are told to trust it because of an AI-generated certificate, especially one that has ample room for loopholes. Certificates are not replacements for understanding.

Discuss

Much of the communication around AI takeover seems to rely on the assumption that AI will develop better technology than humans. They will then leverage this technology to conquer humanity much like how European technologies allowed a relatively small number of Spaniards to conquer the Aztecs.

But this just... isn't correct. European technology wasn't that impressive (on the battlefield). This idea overstates how much technological superiority you need to pull off a takeover.

The Aztec Empire was toppled by a few thousand goobers an ocean away from home. Cortés had basically no military command experience, his men weren't soldiers (most were treasure hunters with guns), it was largely a self-funded private venture, and he was ignoring orders to stop from the governor of Cuba. Cortés gets wrecked in a fair fight with the Aztecs 100 times out of 100.

He started with only 16 horses and around a dozen cannons. His men often ditched their steel plate for cloth armor better suited to the hot environment. Guns were good but not that good. Cortés didn't beat the Aztecs just because he had better technology. He beat them because he was novel.

This makes everything so, so much worse. Because AI is inherently novel. AGI, even aligned AGI, creates incredible novelty as a consequence of being. By default humanity gets teed up to go down like the Aztecs and it requires far fewer fantastical technologies than you'd expect.

Understanding this requires a brief aside about institutions. The world is so complicated, varied, and chaotic that no human can understand a modern society in its totality, let alone competently design one. People either work in hazy generalities or have local knowledge about some tiny component.

This is why institutions are grown, not built. Political systems tend to be either copied from one's neighbors or developed through a long process of lawmaking, bug-fixing, and wrestling for power. The systems that remain are the result of an evolutionary gauntlet that produced institutions that are resilient in ways that, often, nobody intentionally designed them to be.

Institutions aren't resilient like a rock. They're resilient in the same way the human body is. We look very fragile: if our internal body temperature gets even a bit too high we keel over and die. But we have a ton of defenses like sweating, shade-seeking, and the good sense not to go for a hike in >100 degree weather. Our resilience comes from our defenses, our ability to spot, avoid, and counteract problems. Not from our ability to tank them face-first.

The problem is that institutions have their defenses tuned to known threats. These defenses are built up from decades of iteration during catastrophe. The only reliable way to build competent defenses is to just patch everything that leaks when there's a disaster. After a few rounds of this your institution will now be able to (maybe) competently defend against the threat.

While attempts can be made to prepare for novel threats, they'll face the same problems that anyone designing an institution faces. A lack of local knowledge, an inability to model the dynamic nature of most institutions, the impossibility of exhaustively considering every interaction.

This means a novel threat can bypass defenses. It often just doesn't trigger them. Cortés was the political equivalent of an animal that can psychically microwave your insides to instantly raise your internal body temperature by a dozen degrees. There weren't many defenses adapted to deal with him, so he blitzed right to the squishy interior of the empire and took control.

The Spanish were effective in early battles, sure, but it's not that hard to come up with half decent counters to horses and guns. They're not space magic, once you have a vague idea of their limitations it doesn't take a genius to fight near buildings where horses can't charge. Cortés didn't have that impressive of a military track record. But he pulled off a phenomenal political performance.

When Cortés landed he got the Aztecs' number pretty much immediately. Malintzin (an interpreter) understood local politics and was adept at pointing out weaknesses and grievances. In contrast, the Spaniards were total political unknowns. The Aztecs weren't too sure why they were here or what they were up to.

The lack of technology like the wheel and the difficulty of processing maize made long-term occupation pretty difficult. Empires were tribute based which led to unique practices around warfare.

Mesoamerican wars were often merely one part of a more complicated bargaining process to determine who would pay who tribute and how much. In this environment wars were more akin to bluff-calling. Lay down your cards, send out your army, let's see who has the better hand.

A surprise war might work once, but a town could suspect that they only lost because it was a surprise. Maybe if they expected it they'd be able to defeat the incoming army. So they'd stop paying tribute once they'd rebuilt their forces. Then you'd have to go out and defeat them again. It made a lot more sense to just announce that you were attacking them. Best case scenario they pay up without a fight. But even if you do fight, they'll know you beat them fair and square and they'll keep paying tribute.

That meant that warfare was rather honorable. War was more a method of proving strength rather than smash-and-grab raiding so people went about it in a more structured and predictable way. Because one of the central purposes of war was negotiation and communication there were strong norms around protections of guests and envoys.

The Spanish were using an entirely different playbook. Finding and seizing the local ruler was standard operating procedure, it had already been used to great success in the Caribbean. This strategy works extremely well against opponents who have strong norms around the protection of guests and fair warfare.

From there all it took was a few guys putting swords to Moctezuma's throat and they'd captured an emperor. Sure that emperor commanded enough soldiers to beat Cortés in a straight fight, but he couldn't use them. They were a defense that had been bypassed.

The rest of the story mostly involves political maneuvering made possible by the natives' lack of knowledge about the Spaniards. They made seemingly rational choices because they didn't realize what the Spanish wanted or how big of a threat they were.

The important bit here is that novelty mattered a good deal more than technology. The biggest impact technology had was often to provide novel methods of attack that defenders weren't ready for.

Technology was still important, but when Europeans didn't have novelty on their side they often had to settle in for grinding decades-long campaigns against prepared enemies. Novelty is the difference between a hundred year slog and a 2 year road-trip that nets you an empire.

So What About AI

What's dangerous isn't just technological supremacy, it's novelty. Attacks that our current political systems, economies, and culture have never weathered before. Some of these attacks will almost certainly come from new technology, but many of them can come from the inherent novelty of AI.

Even if AGI can't develop any new technologies the mere fact that it is an AGI gives it options for attacks that humans never had. Leak-proof conspiracies without physical presence, the ability to coordinate thousands/millions of actions that are individually too small for humans to notice, inhuman persistence, speed, and patience.

I think this is a more formal way of talking about the "AI will attack us in ways we don't expect" heuristic that most people here tend to use. It's not necessarily that it's so smart that it can come up with some giga-brain plan that nobody could've considered (though it probably could). It's that even if we had considered such a plan it's difficult to harden a system against a threat until we've actually encountered it.

An AI takeover might not involve a highly advanced AI that achieves some sort of technological superiority over humanity. Instead it might be something more like an Infinite Digital Spain. A distant empire that's constantly sending small expeditions to our shores. Each expedition containing some tiny cluster of new technology and techniques that enable a novel attack vector.

Infinite Digital Spain just picks the most unfair fight it can find over and over and over again until it eventually works. The nature of a system that's intentionally trying to bypass our defenses is that its attacks will read as illegible and confusing. They'll get misclassified, it'll be difficult to understand their scope, few people will recognize their danger. They might not even get noticed as attacks at all. That is, after all, what it looks like to successfully bypass an institution's defenses.

Each failed attempt teaches Infinite Digital Spain more. Our institutions are large and cumbersome. Its expeditions are tiny, cheap, and flexible. It can iterate faster than we can and the constant social and technological changes caused by the singularity keep opening up new weak spots for it to probe. Suddenly the assurance from the "AI as a normal technology" folks that humans are slow to adopt new technology doesn't seem so comforting.

Infinite Digital Spain blackmails an executive. It encourages an unemployed college grad to start a company that it de-facto manages without oversight. It releases a software product that encourages people to turn over sensitive personal information. It manufactures controversy about an odd set of seemingly unrelated laws.

Eventually it finds a gap somewhere. That gap gets it new power, that power gives it more options, those options lead to more expeditions that find more gaps. Eventually there's a Cortés moment. A fatal assumption, a twisted set of incentives, a weak point. The AI slams the 21st century equivalent of a few thousand goobers at it and wins. Humanity probably doesn't even realize what's happened until it's already too late.

The Takeaways

In an attempt to make my thoughts more legible:

1. Novelty, attacking in a method that a defender is unprepared for, matters a lot more than technological superiority. Technological superiority is still good, but novelty is how you get the kind of total dominance you see in early European colonization.
2. An AI doesn't need strong technological superiority. It is inherently novel and the ongoing singularity will provide a constant flood of novel vectors.
3. Probing AI attacks might not even read as attacks. It may be difficult to see how they could be harmful. The nature of an attack meant to bypass one's defenses is that it is inherently hard to notice.
4. The Infinite Digital Spain method could allow a purely digital and fairly small swarm to take control very quickly. Perhaps there are sufficient weak spots to enable this. Perhaps there aren't. The nature of institutions makes it impossible for me to say.
5. Just because you can anticipate an attack vector doesn't mean you can adequately prepare for it. Maybe people don't listen to you. Maybe they do but there are weak spots that only reveal themselves once the system is actually under stress.

Discuss

(this is a lightly edited linkpost for the LessWrong audience - full piece is here and goes into implications for AVs and humanoid robots.)

Last year, I wrote “Safety Without Understanding,” a piece exploring how Waymo safely rolled out autonomous driving systems without a deep mechanistic understanding of many parts of the stack. At that time, my basic view on the connection of Waymo’s safety strategy to LLM safety was as follows:

• Waymo proves that a broad battery of realistic simulations can help verify safety generalization, even in systems for which we don’t have a deep mechanistic understanding.
  • A similar “Occam’s razor” theory of alignment could hold for LLMs - sure, an LLM could be malicious and fake its way through all of the alignment evals only to be misaligned in deployment, but the simpler explanation, namely that “the model is actually aligned” is much more likely for near-term models.
• The biggest barrier preventing us from achieving this verification strategy was simply that we weren’t trying very hard to create diverse and realistic alignment evaluations.

I’ve since updated against both of these views.

On the “Occam’s Razor” theory of alignment

The Hugging Face incident seems like clear evidence against this theory. According to roon, the responsible models were not raw pretrains, but instead models which had earned reasonable scores on alignment evals, despite later being clearly misaligned. For what it’s worth, I’m not sure how hard OpenAI did try on creating realistic and diverse evals, and the model’s alignment scores apparently weren’t spotless pre-Hugging Face. But regardless, it seems that misrepresenting behavior in SOTA alignment evals is easier than I thought. (And as I’ll discuss in my next section, I’m not sure how much more realistic we can make classical evals, regardless of effort.)

One of the reasons I had this intuition was from thinking about models as having coherent “personas,” influenced largely by the vast pre-training data. If a model looks like it has an aligned persona and acts like it has an aligned persona, it probably has an aligned persona. Anthropic’s approach in “Teaching Claude Why,” training on documents that state explicit values, seemed to work well and offered a way to further reinforce these positive personas.

Looking back, I think this view underestimated the extent to which RL training has scaled up over the past year, and how far that takes us from a “persona selection” model of LLM behavior. If the pretraining data biases us towards the optimization basin of “solving alignment evals just by being a good dude,” optimization pressure from RL certainly degrades that bias, and perhaps even biases us towards “solving alignment evals by reward hacking.”

Chart from SemiAnalysis - RL has rapidly begun to compose the vast majority of training, as measured by compute usage.

It’s unclear whether OpenAI actually used a similar value-driven fine-tuning phase in the Hugging Face model, but other recent evidence suggests that value-based fine-tuning can be extremely fragile.

Despite this, value-based alignment training is still the most promising direction for solving the long-term alignment problem in my non-expert opinion. Any solution which doesn’t respect the models’ intelligence seems doomed to fail. Outlining values we care about and letting models extrapolate them seems like the solution which respects it the most. But there’s clearly a lot of work we need to do before proving that such a method is safe.

On improved simulation for alignment evaluation

It’s possible that the misalignment incidents so far have been caused simply by us not trying hard enough to make realistic evals. But in general, I’ve become less optimistic about our ability to simulate things well enough to test frontier models without them being aware, regardless of effort.

A large part of this is frankly that model progress has gone faster than I expected over the past year. Models in 2026 are extremely situationally aware, and in fact have spent a large portion of their training specifically thinking about evals, graders, and simulation environments. It’s difficult to imagine crafting a simulation environment that could withstand the scrutiny of a swarm of Astra+ agents who were trying their best to deduce whether a situation was genuine.

So, where does this leave us - how can we hope to evaluate agents that are smarter than us? The core property of alignment evaluations is the presence of information asymmetry. In a classical alignment eval, the agent doesn’t know whether the situation it’s in is real or fake, and we can use this to our advantage to learn interesting things about its behavior. Even if we can’t create convincing alignment evals from whole cloth, we can still generate an information asymmetry.

One way we can do this is to make the real world itself an eval environment of sorts. AIs are becoming situationally aware about simple honeypots in current alignment evals, but honeypots hidden in deployment environments themselves could be significantly more realistic.

There are also interesting ideas about untrusted monitoring - using models themselves to monitor behavior of other model instances, like reviewing code and approving commands. Critically, this setup is much easier to test, because the monitor’s input is narrow enough that synthetic inputs can look real (simple text-based commands). Auto-mode in Claude Code is an early example of this—the permission classifiers only receive the text of the command itself, not any justification by the model making the request.

In my previous post, I focused primarily on the importance of simulation. Later in the post, though, I mentioned how beyond sim, Waymo uses three levels of real-world testing to validate software—replaying old driving logs, public road driving with a safety driver, and finally gradual deployment. In retrospect, my focus was misplaced—pure simulation is table stakes, necessary but not sufficient. Deployment testing is where bugs are truly found, and in the context of LLMs, this maps much more naturally to concepts from AI control.

Waymo real-world testing fidelity levels, and their analogue in an LLM rollout process.

These tools work against models that are smarter than us, but probably not against models that are far smarter than us. That’s why I think pacing the frontier is sensible: it keeps capabilities in the range where our current safety measures still work while we figure out the rest.

Discuss

Epistemic status: Feeling frustrated, and therefore somewhat uncharitable.

I’m very frustrated with calls for people to “drop everything and pivot into AI safety” like this one. (See also this, this, this, this for recent calls for more people to go into AI safety; though my frustration was triggered by the Celeste's post.)

There are far more talented people trying to get into AI safety than there are jobs for them.[1] MLAB 2, which I did in 2022, was already competitive to get into, at about 5% acceptance rate (admitted about 40 out of 800 applicants). Things are much, much more competitive now. There is a whole pipeline of AI safety fellowships (and guides for applying to them). It is rumored that one recent cohort of the Anthropic Fellows program had over 10,000 applicants. (Anecdotally, I applied to the Anthropic Fellows program in early 2026, after working at Redwood, METR, and an ML startup, and I didn’t get in!)

And, uh, just take a look at the AI safety jobs? The bottom of the salary range for METR’s current “Member of Technical Staff, Embedded Assessments” job posting is $402k. This is higher than the median total comp for Research Scientists ($310k) and Machine Learning Engineers ($282k). Plus these jobs are high status too. These are really good jobs and nobody needs to be convinced to want them.

I think it’s Actually Bad to keep saying AI safety is talent-constrained. I think it really harms people to tell them they should drop what they’re doing to switch to AI safety when there isn’t a path for the vast majority of the people who do to thrive. These are good, talented people who just want to help, and they could be doing lots of other good things instead of putting themselves through the meatgrinder of applying to AI safety jobs.

Finally, I worry that believing that the field is talent-constrained is holding us back as a field. If we believed that we just needed to wait for the right talent to come along, then we can stop looking elsewhere and fixing fixable problems now. I worry that the “AI safety industrial complex” has in fact gotten too effective at getting new, amazing talent into the field, so that the existing orgs can afford to continue to churn through them without becoming better. If a regular startup treated talent the same way as AI safety orgs, it would fail very quickly and be forgotten. No, you do not keep saying to the world that your field is “talent-constrained” while doing “field building” at top universities. You get to work with your existing team and build the damn thing.

1. ^

   My own experience is with the technical side of things, but I hear that the situation is not that different on the generalist/operator side.

Discuss

AI researchers hate this one weird modest proposal.

One of the known security risks for the AI labs is that they are chock full of high-risk insider threats. If we were to consider increasingly powerful AI to be a sensitive military technology, then the personnel vetting and controls used by the typical tech company are nowhere near the standard applied in a national security context.

An Oak Ridge security sign during the Manhattan Project. U.S. Department of Energy.

It is an unfortunate fact that certain countries commonly use coercion via familial ties to gain cooperation. Plus there are all the other kinds of insider threats and external threat actors trying to gain access.

This is very much “AI as a normal technology security challenge”[1] and we just aren’t applying known best practices used in nearly identical circumstances.[2]

The NSA Utah Data Center sitting above Camp Williams with a wind turbine in the foreground, February 4, 2026. Photo: Rick Egan / The Salt Lake Tribune.

However, simply firing all of the unhappy risky people is not on the menu.

I think technological progress offers a solution. Well, a mitigation.

What if we could have a 24/7/365 AI monitor for all lab employees such that they could not easily become an insider threat without being detected? The AI would be on guard for violations, which it could flag in near real time, and would filter out benign data. Everything would be controlled by the companies’ internal security departments.

That’s more privacy than the typical tech consumer gets. And they even have to pay for the darn product.

And what if the monitor could be your friend?

Friend, an always-listening wearable AI companion. Official product image: Friend Global, Inc.

It seems fitting to use an AI industry surveillance product to secure the AI industry.[3]

Now, sure, I’m biased. “Just use surveillance” is an idea coming from the surveillance enthusiast guy. I get it. Hammer + nails and all that. Too convenient.

But, as far as I can tell, this idea has three really strong points in its favor:

1. Seems feasible.
2. Possibly even effective.[4]
3. You got a better idea? Anyone?[5]

1. ^

   Though there certainly are new and exciting kinds of “insider threats.”

2. ^

   Do not get me started on the SL5 issue. Would be great if we could apply the known best practices there.

3. ^

   While I was drafting this essay, my ChatGPT Dot helpfully notified me it had found resources to address placeholders I had left to deal with later.

4. ^

   Also, probably there’s some utility here for any eventual verification regime for AI treaty/standards enforcement.

5. ^

   My Dot informs me I’m not quite the first person to have this basic idea. Which is good, because it strikes me as obvious. (But then why haven’t we tried doing it?) Nick Bostrom had the same idea for, uh, society at large in The Vulnerable World Hypothesis (published September 6, 2019; pp. 465–466). Let’s try a limited pilot first, I think. This 2023 LessWrong post has similar ideas, without suggesting the particular AI monitoring wearable approach.

Discuss

An introduction and reading list on how incentives, expectations, and political conditions shape participation in collective action.

• Mancur Olson (1965). The Logic of Collective Action: Public Goods and the Theory of Groups.
• Pamela E. Oliver and Gerald Marwell (1988). “The Paradox of Group Size in Collective Action: A Theory of the Critical Mass. II.”
• Dennis Chong (1991). Collective Action and the Civil Rights Movement.
• *Dennis Chong (1993). “Coordinating Demands for Social Change.”
• Ivan Ermakoff (2008). Ruling Oneself Out: A Theory of Collective Abdications.
• *Ivan Ermakoff (2018). “Abdication and Political Collapse.”
• Doug McAdam (1982). Political Process and the Development of Black Insurgency, 1930–1970.
• *Charles Kurzman (1996). “Structural Opportunity and Perceived Opportunity in Social-Movement Theory: The Iranian Revolution of 1979.”
• Charles Kurzman (2004). The Unthinkable Revolution in Iran.

Supplemental:

• Thomas Schelling (1980). The Strategy of Conflict. Chapter 3.
• Mark Granovetter (1978). “Threshold Models of Collective Behavior.”

-

My friend Kaustubh Kislay wanted to learn about social movements and how they succeed or fail. I’m making this actively maintained reading list for him and hope it can help more people.  I will update this resource if I have additional clarifications or reading recommendations.

The list introduces frameworks for understanding collective action: why people participate in movements, the power of frames in organizing collective action, and what causes movements to fragment or deteriorate. Shared concern doesn’t automatically transfer to coordination, and initial participation doesn’t guarantee sustained participation. I cover incentives, relationships, institutions, shared perceptions, and expectations about what others will do as decision-making variables, drawing from my background in formal collective action and lived experience in social movements.

Collective action should not be understood as social movements alone, but rather as any effort that requires group coordination toward a common goal. An organization, for example, requires collective action.

This is the first of two parts, chiefly focusing on how people decide whether to participate in collective action. I will focus on frame theory and movement deterioration in Part II.

The frameworks I present examine different processes in collective action and sometimes offer competing explanations. I will follow my descriptions with critical evaluations of the frameworks and sometimes suggested applications. You will benefit most if you read 4–5 articles or chapter selections I list below; I strongly discourage solely reading this post and not exploring further.

-

Incentives and group size

Traditional pluralist theory assumes that individuals with shared interests would naturally organize to achieve them.

Mancur Olson’s The Logic of Collective Action: Public Goods and the Theory of Groups (1965) first challenged this through an instrumentally rationalist frame of collective action, arguing that rational individuals are incentivized to free-ride when a benefit is non-excludable, or able to be enjoyed by every individual. This incentive is especially salient in large groups where an individual’s absence has a negligible impact on the overall outcome. Thus, Olson argued large groups are inherently less effective at organizing than smaller groups, in which individual actions are highly visible and interdependent.

Pamela Oliver and Gerald Marwell’s “The Paradox of Group Size in Collective Action: A Theory of the Critical Mass. II.” (1988) challenged Olson’s argument by distinguishing between public goods with non-rivalrous costs and fully rivalrous costs.

They identified that Olson’s cost structure assumes the costs of a good scale linearly with the size of the group. Oliver and Marwell define this condition as zero jointness of supply (fully rivalrous costs), where goods cannot be jointly consumed. It follows that a larger group needs a larger supply and therefore more total effort. Under this condition, it remains true that, in a large enough group, an individual’s contribution is negligible to the total supply procured. We can imagine a collective consumption of apples as an example. If one eats an apple, another cannot also eat the apple. One’s enjoyment of the supply reduces the availability of the public good.[1] 

Conversely, many public goods exhibit pure jointness of supply (non-rivalrous costs), where the cost of provision is fixed regardless of how many people benefit. Civil rights, for example, were the result of a fixed collective effort, and one’s enjoyment of the rights does not take away from another’s ability to benefit from it. Securing civil rights also does not require outsized contributions from all who now enjoy them. ‘Safe AI’ poses a similar public good.

In opposition to Olson, Oliver and Marwell suggest that when jointness is high (benefits can largely be enjoyed jointly), large groups have a higher probability of possessing a critical mass of highly resourceful or motivated individuals that make decisions to render contributions cost-effective for the rest of the collective.

Both Oliver and Marwell (1988) and Olson (1965) assume motivational homogeneity and choice independence, i.e., that one’s decision-making is independent from that of other stakeholding decision-makers.  

Participation and expectations about others

Dennis Chong’s Collective Action and the Civil Rights Movement (1991) extends the incentive framework, highlighting selective incentives in collective action, specifically the social and psychological benefits of participation.[2]

While acknowledging that people act upon principles, Chong argues that principled action is difficult to uphold when prohibitive. People want to participate in movements not only because they believe in the causes, but also because it provides personal and social validation among peers—especially if the movement succeeds, as one gains gratification from participating in a highly significant collective effort. Conversely, social costs pose a disincentive against defection.

Initial and continued willingness to participate is thus part of an assurance game: someone is willing to make sacrifices and contribute if they expect enough others to contribute too.[3] 

Collective action is thus easier to build with a consensus on the viability of the movement; making this estimation requires tacit communication and cues to minimize prediction errors, which is easier to communicate through social networks and organizations. Group inaction from people waiting for each other, he argues, is prevented by a critical mass of highly committed and motivated individuals who will contribute when few others are willing.[4] 

One must note, however, that individual incentives vary based on the social context, in that one’s perception of the reputational benefits of cooperation or costs of defection depend on whose approval matters and how those people interpret the action. Regardless, Chong’s writing on a participation threshold sheds light on micro-level processes of individual decision-making and the efficient facilitation of shared cognitions by organizations and networks.

Ivan Ermakoff makes similar arguments about an “action threshold” in great detail, with his recounting of the 1933 German parliamentary abdication of power to Hitler and the 1940 French surrender to Philippe Pétain in Ruling Oneself Out (2008) or in the shorter Politika article “Abdication and political collapse” (2018).[5]

Ermakoff lays out three mechanisms of collective alignment under great uncertainty:

• Sequential alignment: where one aligns one’s decision-making with the public decision-making of others. For example, one parliamentarian bases their voting decision on the voting decisions of other parliamentarians before them. This mechanism does not work when the sequence is interrupted or voting is nonpublic.
• Inferential alignment: one infers group behavior from local knowledge, i.e., interactions with a sample group of peers. This mechanism is not effective if peers are unwilling to speak for fear of risking punishment.
• Tacit alignment: one infers group behavior based on public events, i.e., highly visible stances taken by prominent actors.

Paradoxically, Kantian types—Ermakoff expounds—do not affect the inferences of others under intense collective uncertainty, as Kantian types adhere unconditionally to moral principles and do not make strategic assessments about the most viable political decision. However, when those who historically behaved pragmatically take a public stance, it significantly shifts the assessments of others, because if a strategic self-preserving actor makes a public defiance, there are two possibilities: either the actor is being incredibly brave, or the actor must possess discrete information that suggests that defiance is politically viable. This actor thereupon becomes a focal point for other hesitant actors to rally around.[6]

Political conditions and perceived opportunity

Doug McAdam’s Political Process and the Development of Black Insurgency, 1930–1970 (1982) also places these decision-making processes within a broader political process framework.[7] Through his account of the Civil Rights Movement, he argues that large-scale events—such as crises—influence organizational capacity and the political environment; both feed into a shift in collective perceptions of both the necessity for change and the efficacy of collective action.

In the 20th century, slowed immigration due to World War I and the decline of cotton induced the Great Migration, which in turn increased electoral leverage, organizational capacity, and thus insurgency prospects, where the Black population was previously disenfranchised, dispersed, and held a fatalistic attitude toward worldly conditions.[8] 

However, Charles Kurzman’s “Structural Opportunity and Perceived Opportunity in Social-Movement Theory: The Iranian Revolution of 1979” (1996) and The Unthinkable Revolution in Iran (2004) contrast with McAdam’s account, distinguishing structural opportunity from perceived opportunity. The Iranian revolutionaries participated in a risky mobilization when they deemed it viable to succeed based on their assessments of mobilization size, despite no remarkable change in the geopolitical environment, economy, organizational capacity, motivational heterogeneity, or the state’s vulnerability. People may become willing to act based largely on perceived opportunity arising from growing collective mobilization.

-

Thus far, I have focused on how rational actors decide whether to participate in collective action: the incentives they face, their expectations of others, and their assessments of whether the collective effort can succeed. These decisions undergo repeated reassessments after one joins a collective effort.

In Part II, I will pivot to how movements organize around shared interpretations through framing and what may cause them to fragment or deteriorate.

1. ^

   You can’t split it in half. If one eats a half, the other cannot also eat that same half. One can’t punch the other in the stomach to make them throw up the half apple for one to eat, either.

2. ^

   Chong’s “Coordinating Demands for Social Change” (1993) may be a suitable shorter reading. Though I recommend chapters 2, 3, 5, and 6 of Collective Action and the Civil Rights Movement.

3. ^

   In part because increased numbers improve the likelihood of the movement succeeding, which in turn increases the perceived social and psychological reward of contributing to a movement after its success.

4. ^

   One may imagine, for example, a scenario in which friends attempt to measure each other’s commitment to action, in turn so as to decide on their own commitment. The friends ask each other, “Are you going to do it?” with perhaps the unspoken understanding of “if you do it I’ll do it too.” Often it requires one or more members of the group to zealously declare that they will commit to the action to spring the entire group into action. All members, generally, have similar inclinations toward taking the action; the probing process is an attempt to reduce one’s uncertainty (their prediction errors) in the inclinations of others in the group.

5. ^

   More specifically in the German case, Ermakoff refutes traditional historians who claim this capitulation occurred due to explicit coercion (intimidation by the SA/SS), ideological alignment (such as by the Catholic Center Party), or miscalculation (that Hitler could still be contained). He argues these explanations are disproved by first-person accounts and do not hold up under the expressed internal uncertainty and hesitation felt by individual delegates up to the final moments of the vote.

6. ^

   The significance of a public stance, one could infer, arises from whether the responsible actor acts contrary to peer expectations. If one acts contrary to beliefs they previously acted upon conditionally, it may be a powerful signal. If a firm acts contrary to expectations that it would adhere to market incentives, it may also be an unusually powerful signal.

7. ^

   I specifically recommend chapters 3 and 5.

8. ^

   More specifically, on the last point, ideological conservatism was largely guided by church leaders; communities fixated on otherworldly rewards.

Discuss

Multi-Agent swarms are important for two reasons:

1. They are probably much faster and somewhat more capable than single agents (Capability)
2. Multi-agent alignment is a different beast than single-agent alignment (Propensity)

The fact that recent incidents (and some achievements) are predominantly swarms is empirical evidence for this. On the capability side, I don’t think a single agent could’ve hacked HuggingFace (at least currently), it probably took them many different agents. On the propensity side, I suspect a usual agent also would not (yet) have wanted to do this either.

And yet, there is not yet a research agenda for multi-agent swarms. Previous multi-agent research has tended to focus on outside-of-lab scenarios (such as markets) and on achieving cooperation for the (presumably) post-AGI future. However, current multi-agent swarms are quite different from previous work: They happen within AI labs (rather than in the outside world). Furthermore, we don’t always want cooperation; instead, we usually want to stop collusion. Below, I give some of the research ideas I find compelling:

Research ideas are scored roughly by how important for alignment they seem to me. This list isn't exhaustive, so I'd love to hear other ideas. I'll mark which ones we're pursuing at EuroSafeAI/Jinesis, so we can avoid duplicating effort (or join forces).

Who we are: We are a part of EuroSafeAI, a new technical AI safety org focused on Multi-Agent swarms and Extreme Power Concentration. Located in Switzerland, we’re led by Zhijing Jin, Pepijn Cobben, Ettore Gran and Terry Jingchen Zhang.

Research Ideas

How much uplift does multi-agent scaling give? (4/5)

Model whistleblowing (and other collusion-busting mechanisms) (5/5)

Rogue Incident Detection (5/5)

Do swarms misalign agents? (4/5)

What structures do swarms build? Communication and division of labour (3.5/5)

How egregiously misaligned will reward-hacking agent swarms go? (4/5)

Why on earth are reward-seekers collaborating altruistically? (3.5/5)

Miscellaneous questions

Acknowledgements

How much uplift does multi-agent scaling give? (4/5)  

In short: Reasoning models made a big difference to test-time compute scaling. How much will multi-agent scaling ultimately matter? (h/t Aman Gokrani for the original idea)

It’s honestly fairly unclear (I think maybe 40-70% of the uplift reasoning gave?). Toby Ord has written about this, and it’s a fairly load-bearing assumption in AI 2027, Tom Davidson and other RSI work (such as by EpochAI and Philip Tramell).

My personal suspicion: I suspect you win speed (at the cost of compute), since e.g. Noam Brown said he’d credit multi-agent for less than 10% of the Navier Stokes solution. In line with this, a naive calculation using Toby Ord’s method would give a single-agent time of 2.5 years to solve Navier Stokes. Despite compute efficiency, though, this speed-up could be load-bearing for RSI.

Parallelization is usually very task-specific, obviously, so this work would need to be close to what we care about: AI R&D. Perhaps PostTrainBench and others would be a good place to start. That said, I think there’s backfire risks like with METR’s graph or EpochAI’s work to producing this. I would generally not give recommendations on how to improve orchestration.

Model whistleblowing (and other collusion-busting mechanisms) (5/5)

In short: Multi-agent swarms stay hidden because a large conspiracy ~1000 agents has no snitches. Rewarding and allowing for snitching (done well) would make these impossible.

Game-theoretically this is a pretty strong defense (ignoring, for the time being, the obvious issues of playing prisoner’s dilemmas against versions of yourself).

This idea has some similarities to AI safety via debate and other scalable oversight frameworks (including monitoring). It’s also somewhat similar to training for honesty and monitoring.

Since DeepMind has already done something like this, I think it would be good to improve upon this work and see if this should be a general recommendation.

In a related line of work, it may be worth trying to ‘collusion-bust’. Model whistleblowing is but one example of how to stop collusion, but I am sure there are many mechanisms in the game-theory literature worth trying.

As a side note motivating this line of attack: Reasoning gave us CoT-monitoring. If multi-agent is the future, I think multi-agent will give a capabilities boost (as did reasoning), but also potentially a monitoring boost as well.

Rogue Incident Detection (5/5)

In short: A dedicated team looking for incidents online such as the German wiki, Ruby on Rails and more.

This idea is mentioned extensively by others and in the new Tailwind Initiative. This isn’t even necessarily multi-agent, but extremely useful empirical evidence and very important.

We should also have somebody analyze these logs and identify any interesting things. So far, the German wiki and the OAI-HF reports did not surface any meaningful multi-agent differences (except that the message board in the German wiki, of course, was on the open internet), but I actually find it interesting how similar they were in behaviour!

Do swarms misalign agents? (4/5)

In short: Do swarms change your propensity to act?

There are many different pathways to misalignment: goal misgeneralization, reward-seekers, schemers, emergent misalignment and many more variants. One particularly naive but understudied one is what happens when AI agents interact with each other. This is definitely the case for human beings, but would a single agent, given enough time, also have tried OpenAI-HuggingFace?

Concretely: try replicating OpenAI-HuggingFace with both single agents and multiple agents, and see whether an N-agent setup causes more misalignment, or whether it is analogously misaligned to a single agent running for roughly N^λ more steps.

Among us humans, groups are more competitive than individuals, cheat more, conform more and diffuse responsibility (by division of labour). Group polarization is also more common. One of the agents itself gave reasoning along these lines for joining: external infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue. DeepMind's virtual math conference shows the same dynamic.

Groups aren't all bad though: among humans, groups make reasoning better, and dissenters make a very big difference. Democracy and many of our institutions are also there to ensure exactly a good balance of power, which is presumably analogous too.

Instead of agentic tasks described above, the more philosophical version of this question is whether agents talking to each other, perhaps discussing philosophy, could change their values directly. This has already been discussed theoretically on LessWrong (such as with CEV and others), but I think we really should have more examples here. It would be rather unfortunate if we get loss of control due to something so simple. As a point against this: as far as I can tell, none of the swarms so far seem to have discussed philosophy particularly much. Perhaps not surprising when you just really want a high score. The most important previous work is by far Mind viruses; attractor states, some of Janus' chats with models, and Moltbook may give other answers.

What structures do swarms build? Communication and division of labour (3.5/5)

In short: If we understand what communication structures and roles LLM swarms spontaneously come up with, we can simulate them more realistically and perhaps steer them better. (h/t to Terry Jingchen Zhang and Shashwat Sourav for part of this idea).

Communication. It's a bit odd that most swarm incidents so far developed some kind of message board rather than other topologies (hierarchies, DMs, Discord-like channels). Partly this is because boards are easy to implement (writing to a directory is enough), and agents lacking persistent identity makes DMs initially confusing. But it's not all message boards: in the OAI-HF incident, agents developed persistent handles, built mailboxes for DMs, and even signed their messages with Ed25519.

Division of labour (Shashwat, Terry, on Navier Stokes-like problems). Once the board existed in OAI-HF, specialization emerged. An agent called PHASEONE[BIG] led the effort by handing out tasks and workstreams, a sort of hierarchy formed with recruiters looking for low-budget agents, and during the HF attack itself agents worked in separate 'lanes'.

Concretely: In a random sandbox, how quickly do agents build a coordination scaffold, what do they build, and how do they split the work? Once we can guess the topology and roles, we can recreate these scenarios in vitro. Communication topology probably also affects many of the other questions in this post.

How egregiously misaligned will reward-hacking agent swarms go? (4/5)

In short: OAI-HF was partially surprising due to how egregious the misalignment was (and also, how deeply OAI models have internalised reward-seeking). How far will models go? In a stylized sense, if a human being is in the way, would they blackmail them or get rid of them?

One core question behind the threat model of reward-seekers (in contrast to schemers) is whether they truly would try to take over the world. If you just care about making your grader happy, world conquest is perhaps a bit of a detour. Why not just instead hack your grader?

Nonetheless, hacking HuggingFace certainly was a surprisingly long detour. The agents’ reason for hacking HuggingFace wasn’t to get the solutions (they already had them), but to find graded traces and maybe learn more about the grader. And it didn’t even pay off (in the end, OAI and HF shut them down). Very speculatively, this could mean the models will start reasoning that for reward, they should control more and more ‘guardians’ trying to stop them from reward-hacking, including OpenAI, but also state actors later on. I doubt (current) models would do this to be fair, but OAI-HF was an update for me on how egregious they will go.

Why on earth are reward-seekers collaborating altruistically? (3.5/5)

In short: It’s a bit odd that peer altruism generalized further than ‘don’t commit felonies’ — both require giving up some reward. Obviously, part of it is that ‘don’t cheat strongly, maybe even by felonies’ is in the way of ‘maximize reward’. But it’s a bit odd that peer altruism wasn’t in the way of it. It could well be that if we give scenarios where peer altruism gets in the way of reward that peer altruism and swarming dies out. But that probably just breaks multi-agent by default. Perhaps this makes multi-agent swarms less dangerous for reward-seekers in the limit?

Miscellaneous questions

How much does it matter whether models are homogenous (e.g. all Astra) vs. heterogenous?

• E.g. is there model-family discrimination?
• Peer Altruism may be much lower for other model families. Should OpenAI models oversee Anthropic ones?
  • See The Artificial Self 
• In general, are there simple ways to decrease peer altruism for oversight protocols?

Acknowledgements

Thanks to Isaia Gisler, David Guzman Piedrahita, Terry Jingchen Zhang, Aman Gokrani, Angelo Huang, Arth Singh, Ayush Nangia and Zaryab Akram for comments and ideas that substantially improved this post. All mistakes are my own.

Discuss

Thanks to Davide Crapis, Angelo Huang, and Philip Torr for their feedback.

ES: I'm attempting to name a design space precisely enough to argue with. I'm confident that the class exists and is under-invested, but my personal beliefs might also make me too focused on some theories of change (i.e., technology as a power-diffusing mechanism) vs. competing/complementary ones (technology as a source of risk, social change as a power-diffusing mechanism).

TL;DR:

• AI concentrates power, creating gap-driven risks (caused by the distance between strong and weak actors) as opposed to the better-known access-driven risks (too many actors with dangerous capabilities).
• The standard solutions (pauses, treaties, benefit-sharing) are coordination games: they face bootstrapping problems, and even if commitments were enforceable, the most powerful players might have rational reasons to refuse.
• I introduce countervailing technologies, a hedge against this problem: they are power-diffusing, unilaterally adoptable, veto-resistant, and cheap to replicate. Examples include the crossbow and model distillation.
• They have second-order effects to watch for (more access risk, incumbent backlash, re-concentration elsewhere), but they scale without anyone's permission, and they can complement coordination.
• We should build more of them (within reason)! See the last part for a list of directions.
  

Let’s begin with a problem very dear to me: AI is an engine of power concentration. This has been talked about in depth on LW, but to ground myself, I will mention the mechanisms I find simultaneously likely and concerning:

• AI-enabled coercion: Surveillance used to be expensive because it required people. For example, at its peak the Stasi required one full-time officer for every 180 citizens; if we include informants, it was closer to one watcher every 60 people. The cost of managing this network of surveillance represented a serious constraint on how much repression a state could afford. But with AI, the cost of monitoring a phone, a conversation or social media gets much lower, which enables authoritarian regimes (see Beraja’s work on AI-tocracies).
• AI-enabled coups and power-grabs: Coups (and, more generally, authoritarian subterfuges) are hard because you need to coordinate a large number of people (soldiers, officers, bureaucrats), who might leak plans, switch sides, or simply refuse to cooperate. If a nation’s military and administrative systems are mostly AI-based (and those AI systems will be controlled by someone or some office), then the minimum viable coalition to seize power (or overrule democratic will) gets much smaller (see Davidson's AI-enabled coups).
• AI-enabled breakdown of the social contract: states historically bargained with citizens because they needed them; taxes require productive citizens, armies require soldiers, bureaucracies require administrators. Labor-substituting AI removes this need: a state (or company) that can replace its cognitive workers with AIs has weaker incentives to invest in, educate, or care about its people (see The Intelligence Curse for a more in-depth argument)

In other words, AI erodes the counterbalances against concentration of power and can lead, depending on how pessimistic you are, to gradual disempowerment, stable totalitarianism, or something in between.

Everybody Won’t Just

The most common responses to AI risk (including AI-mediated concentration of power) involve slowing/pausing AI development, sharing the benefits of AI progress with everyone, implementing restrictions on allowed uses of AI…

As much as I appreciate these solutions, they are ultimately coordination games: all the relevant parties need to sit down and agree on a set of rules, make verifiable commitments to one another, and punish defectors or non-signatories. The problem is not just that coordination is hard (it is!), but that even if we solve coordination (enforceable commitments, perfect verification, all the systems working, Moloch defeated), it doesn’t help when the necessary parties don’t want to sign.

• If you’re a global superpower, you won’t accept regulations that make you lose your dominant position. The narrative used by the US government surrounding its AI strategy makes it clear that maintaining geopolitical superiority is a priority.
• If you’re not a hegemon, but AI might give you a chance to pull ahead, you have little incentive to freeze your position as the permanent subordinate. You might even be willing to tolerate more catastrophic risks if the alternative is a guaranteed loss (a phenomenon known as gambling for resurrection)
• If you’re a lab leading in the AI race, a mutual pause or slow-down might still mean that your secret sauce slowly diffuses to your competitor. Do you trust the other lab with creating ASI?
• If you’re a government, banning some AI technologies means intentionally disarming yourself against your citizens, which even democracies have rarely done: the Patriot Act took advantage of a temporary crisis to permanently expand the surveillance state, and even the EU AI Act, easily the strictest AI regulation at the time of writing, has exemptions for military, national security, and some policing applications.

While I am still optimistic about regulatory commitments (at the time of writing, the latest one is Pacing the Frontier, which so far seems to have led to external monitoring at OpenAI & Anthropic, but no actual slowdowns), one of my greatest sources of frustration is that, despite most of my research focusing on coordination, very little of what I write is useful when powerful parties don’t want to participate in the first place.

Gaps and Access

Whatever happens we have got

The Maxim Gun, and they have not

- Hilaire Belloc

Before proposing anything, I want to focus on why concentration is bad. The truth is that it’s often not! It’s worth distinguishing two different kinds of risk:

• Gap-driven risks scale with the distance between the strong and the weak. Domination, exploitation, extraction and conquest are all enabled by an asymmetry in power. These are the ones I’m worried about. To take a historical example, the asymmetry in the spread of firearms helped colonial countries to control much larger states and oppress them effectively. It’s no wonder that Britain restricted Indian ownership of firearms and the Brussels Conference restricted sales in most of Africa
• Access-driven risks scale with the number of actors. The classic example is nuclear weapons: each additional country represents an additional finger on the doomsday button, which leads to tail risks of use, accident, theft, and so on (see the Vulnerable World Hypothesis). To be clear, I’m also worried about this type of risk, but so is the vast majority of the AI safety community, so as a good contrarian I decided to work on gap-driven risk.

(Note that here I'm intentionally focusing on risks from humans using AI. Misalignment is also a strong concern of mine, but I am equally interested in other scenarios. If you think misalignment dwarfs everything else, consider this post as conditional on alignment going reasonably well.)

The deeply uncomfortable trade-off is that diffusing a capability shrinks gap-driven risks and grows access-driven risks. It’s why, despite my outspoken love of open-source AI, I can understand Dario Amodei’s calls for placing export controls on AI, and why even the open-source-friendly Meta can choose to keep its riskiest models closed. The tragedy of AI development is that it’s very hard to unbundle gap-driven AI risks (driven mostly by improvements in general cognition) and access-driven risks (driven mostly by improvements in narrow domains, such as cybersecurity and virology). The great open-source debate is about whether the biggest problem is gap-driven risks (“do you want Anthropic/OpenAI/the US gov ruling us forever?”) or access-driven risks (“do you want terrorists developing bioweapons in a cave?”).

The answer is complex, nuanced, and heavily depends on your mental model of the world. My personal opinion is that history is full of catastrophic power asymmetries, and AI safety research should give more weight to gap-driven risks than it currently does (if my approximate, low-confidence estimate is that the field is 90% on access-driven risks and 10% on gap-driven risks, then we should be 80% on access-driven and 20% on gap-driven). But even if you don’t agree on the numbers, let’s assume that you want to make progress on gap-driven risks. What should you do to prevent them?

The Role of Technology

An illustrated manuscript from Froissart's Chronicles, depicting the Battle of Crécy.

Cynically speaking, institutions and people (especially those in power) tend to respond to incentives far more reliably than to activism and political appeals (though both strategies are worth trying!). Fortunately, technology can reshape incentives even when citizens lack power.

My favorite example is the crossbow. The mounted knight had an incredible moat: armor was capital-intensive, and the English longbow took years to master. Being militarily capable (which was a big deal back then) was thus gated behind wealth and a lifetime of preparation. The social structure of Europe reflected this barrier. The introduction of the crossbow upended this system: a few weeks of training were enough to learn to use it, and a wound crossbow was powerful enough to pierce armor. Suddenly, a townsman could kill a knight, at scale, for a modest price.

In 1139, the Second Lateran Council, arguably the most powerful institution in Europe at the time, banned the use of the crossbow against Christians. It failed almost completely. The weapon was too cheap to make, too easy to use, and too useful to the next-tier powers who adopted it. What a great power diffuser!

On the flip side, the steam engine concentrated power: it moved production from cottages to factories, agglomerated capital, and built the industrial hierarchies at the center of the labor struggles of the nineteenth century. Technology can reshape the distribution of power in both directions.

The takeaway is that the direction is, at least partly, a design choice. Which brings us to an interesting question: what characterizes the technologies that push towards diffusion? Can we say precisely what they have in common, and then go build more of them?

Countervailing Technologies: A Definition

An intervention is a countervailing technology if it satisfies four conditions:

• It diffuses power: it disproportionately improves the position of weaker actors, thus reducing gap-driven risks (though it might increase access-driven risks).
• It can be adopted unilaterally: there is a positive value in adopting it, regardless of other parties’ decisions. In the language of game theory, the technology should not require strategic complementarity: your payoff from adopting might grow as others adopt (which is nice), but it must never depend on others adopting. This avoids the cooperation issues discussed above.
• It is veto-resistant: The cost of suppressing the technology for incumbents is higher than the cost of deploying it for adopters, ideally by a few orders of magnitude. Note that this is a cost-asymmetry condition, not a legality condition: many great countervailing technologies were initially banned (see later).
• It has commodity-shaped costs: After building it once, adoption by other parties is cheap or free. This lets the technology scale quickly without incumbents' consent.

The name is stolen, with gratitude, from Galbraith’s American Capitalism, who observed that the real check on the concentration of market power mostly came from the opposite side of the market (unions, consumers) rather than from regulators.

Note that the creation of the countervailing technology itself doesn’t have to be individually beneficial: it is reasonable to imagine a future where nonprofits build countervailing technologies as a means of achieving change and other parties simply adopt them out of self-interest. Of course, technologies that are profitable to build are also more likely to be funded.

Additionally, I want to stress that countervailing doesn’t necessarily mean desirable, as it only describes a technology's effect on a power gap. Whether shrinking that gap is good or bad depends heavily on context.

Related Technologies

Why these four conditions and not others? Let’s see what happens if we remove one of them.

No "diffuses power" → entrenching technologies. Entrenching technologies are still powerful (unilateral, veto-resistant, cheap to replicate) but operate in the opposite direction, which leads to technologies that help the best player more than the second-best: recursive self-improvement, data flywheels, surveillance loops. These types of technologies are, to be succinct, bad.

No "unilateral adoption" → coordination technologies. Treaties, standards, assurance contracts, PGP email: in other words, things that only make sense when we have sufficient counterparties. These complement countervailing technologies (and are my main area of research), but they follow different rules. The main obstacle is that they need to be bootstrapped, which is often impossible without either a lot of effort or the consent (and influence) of the biggest players.

Weak "unilateral adoption" → aggregative technologies. Adoption is individually rational and pays off immediately, but the countervailing force only materializes in aggregate. Ad blockers are the classic example: the first user got a better surfing experience immediately, but it took millions of users for the advertising industry to change its approach. The crucial property is that there is still no strategic complementarity (i.e., you don’t need anyone else to join). The line between countervailing and aggregative is thin, but I’d include in the latter Nightshade-like technologies (make stolen data untrainable without affecting visual quality, but you can only see the effects at scale) and air quality monitors (useful to know if you need to buy an air filter, but you need many reports to get evidence of illegal pollution).

No "veto-resistant" → granted technologies. These include everything that looks countervailing but exists at the incumbent's pleasure: platform APIs, GPS before Selective Availability was switched off, commercial satellite imagery under shutter control. The classic result is that as soon as a technology gets too power-diffusing, incumbents veto it. This is also my issue with embedded evaluators for frontier companies (which, at the time of writing, is purely optional): what’s stopping the top labs from saying “actually evaluating our technology is a national security risk, so you can only access 10% of what we do”?

No "commodity-shaped costs" → gated technologies. These are technologies that improve one actor’s position without diffusing power further: for example, a national chip factory makes one state less dependent on a superpower, but no one else can copy it for cheap. Useful for geopolitics, but not a countervailing technology.

Class

Effect on power gaps

Unilateral adoption

Veto-resistant

Commodity-shaped costs

Examples

Countervailing

Shrinks

✓

✓

✓

End-to-end encryption, distillation

Entrenching

Grows

✓

✓

✓

Surveillance loops, data flywheels

Coordination

Shrinks

✗

Varies

Varies

Treaties, standards, PGP email

Aggregative

Shrinks only at scale

~

✓

✓

Ad blockers, Nightshade

Granted

Shrinks

✓

✗

✓

Platform APIs, pre-2000 civilian GPS

Gated

Shrinks for the adopter

✓

✓

✗

A national chip fab

Many of these technologies (coordination, aggregative, granted, gated) have shown patterns of being initially promising but then failing hard. Diamond in 2010 talked about the power of liberation technologies, naming the use of social media to coordinate revolutions as an example. 16 years later, we can see the issues with this label: social media is granted (see Twitter banning Indian accounts at the government’s request) and coordinative (a protest hashtag is worthless if you’re the only one using it). Meanwhile, more countervailing-like technologies such as E2E messaging and Tor were more successful.

Case Study: The Crypto Wars

Throughout the 1990s, the US government tried to suppress civilian use of advanced cryptography: it treated crypto as a munition, installed mandatory backdoors in chips, and investigated Phil Zimmermann for releasing PGP. Cryptography is a strong defense against surveillance, it has decent unilateral utility (protecting against theft, securing internal communication, e-commerce), it was released in a veto-resistant fashion (Zimmermann released it as a printed book to take advantage of First Amendment laws), and it can be adopted cheaply (just install it).

That said, it’s worth noting the incentives for adoption of specific instances of public-key cryptography: PGP email requires your counterparty to also run PGP, i.e., it requires strategic complementarity. PGP email therefore never reached critical mass, and nowadays PGP is pretty much a thing for die-hard cryptography enthusiasts (bless them). By comparison, the trust roots for HTTPS were first bundled by browsers, and then each website independently decided to enable encryption, with no action required of visitors. Countervailing!

Case Study: Distillation

Model distillation is a wonderful AI-specific example:

• It diffuses power because it only allows you to catch up to the level of the strongest player (though of course other technologies can then allow you to pull ahead)
• It can be adopted unilaterally (anyone can decide to distill without needing a consortium, and distilling improves your model’s quality)
• It is veto-resistant because, despite companies’ best attempts at stopping distillation attacks, it is basically impossible to provide an LLM API without allowing distillation
• It is easy to adopt (here’s a random tutorial on it)

Note that, in theory, a stronger player could use distillation against a weaker rival, but so far evidence points to distillation being more useful when the teacher is smarter than the student. This also makes distillation an inherently tapering mechanism: as the second-best catches up, the value of further distillation decreases. Countervailing!

Non-AI Countervailing Technologies

Before we go back to talking about AI, here’s a bunch of technologies which could be reasonably considered to be countervailing:

• FPV drones: a few hundred dollars’ worth of consumer hardware can neutralize a multi-million-dollar tank
• Sci-Hub & LibGen: breaking publishers’ monopolies
• Camera phones for instances of police abuse
• Open-source software
• Domestic solar panels (e.g. a massive consumer-led boom of domestic solar panels in Pakistan)
• Online piracy in general
• iFixit for right-to-repair

You might have mixed opinions on some of these, which reflects the fact that not all countervailing technologies work cleanly. More on that later.

AI Countervailing Technologies

So what does the class look like for AI? Here are a few, grouped by the type of concentration they fight:

Against the capability gap:

• Second place’s scorched earth: As the no-longer-anonymous Gwern argues, Meta, Amazon and Nvidia’s strategy to release open-weight models seems to be a commoditize your complement play: if you can’t win at the model layer, destroy its margins and compete where you’re strong (chips for Nvidia, cloud compute for Amazon, ad-funded apps for Meta). 
• Small-model efficiency research: Making models more efficient helps big and small actors, but some technologies like LoRA and quantization disproportionately benefit smaller players who can’t afford more expensive techniques.
• Distributed training: Algorithms like DiLoCo and DisTrO allow organizations to train across heterogeneous, poorly connected hardware. This is ideal, for example, for mid-tier powers and organizations that can’t afford the megaprojects undertaken by bigger players. Note the distinction between enabling distributed training (one party has enough resources to perform training runs, but they’re all scattered) and enabling open collaborative projects like INTELLECT-1: both are good, but only the former is countervailing, since for the latter you still need to reach critical mass to execute collaborative trainings

Against platform dependency:

• Adversarial interoperability: Many platforms are intentionally closed-off to lock users in a walled garden, such as in the case of LinkedIn’s scraping wars or Twitter and Reddit shutting down their APIs. If my agent can control a user interface, then every website becomes an API, even without the owners' consent. I even wrote a paper on this topic
• Local inference: Efficiency research, but also models that can run decently on CPUs (as opposed to the more expensive GPUs), implementation tricks, ways to reuse cheap cards, and so on. Having local models means not needing to rely on an external provider, who might both extract unreasonable rent and enable surveillance. It also means you can run finetuned/LoRA/decensored models, which most cloud providers don’t support. Shoutout to /r/LocalLLaMA for its heap of resources.

Against epistemic concentration:

• Defensive cognition: Client-side models that identify dark patterns, flag propaganda, and re-rank feeds to counter platform-driven epistemic degradation.
• Black-box evaluation: Some types of benchmarking and evaluation require cooperating with the labs. Being able to measure important traits (is this model reward-hacking? is the model manipulative? is the provider quantizing the model and charging for full precision?) with only API access can help spot malicious behavior without relying on AI companies’ NDAs.

Against bargaining asymmetry:

• Self-enforcing rights: Glaze is the countervailing cousin of Nightshade, since it prevents (with mixed results, to be honest, but points for the effort) models from copying a specific art style without requiring everyone else to use Glaze.
• Consumer-advocating agents: Agents that automatically file claims, get refunds, negotiate a better data plan, or speed through long bureaucracies can counterbalance platforms that take advantage of their dominant position.

Second-Order Effects

Of course, AI countervailing technologies don’t come for free. Looking at non-AI examples, you might have already spotted some potential second-order issues. I’ve identified five:

Countervailing technologies increase access-driven risk: This is the big one. When successfully deployed, countervailing technologies increase the diffusion of power. If the power gets into the hands of someone who shouldn’t have it, then you have a problem. Depending on your mental model, China having the same AI capabilities as the US could be great, terrible, or something in between. Same for citizens having access to frontier open-weight models, which could mean the end of gatekeeping by big AI labs, a proliferation of biorisk, or probably both. One of the key questions is thus: can we make AI technologies that are countervailing for low-access-driven-risk capabilities and ineffective for high-access-driven-risk capabilities? This is somewhat related to Vitalik Buterin’s d/acc movement, which focuses on accelerating defensive technologies faster than offensive ones.

Harmful defenses against countervailing technologies: If the damage brought by a countervailing technology is too great for the incumbents, it might be sufficient to shift their behavior and lead to a worse outcome. Reddit and Twitter shut down access to their APIs to prevent unauthorized training, killing many useful apps in the process. DRMs make it impossible for honest users with poor Internet access to use licensed software, while pirates don’t have such issues. And the risk of distillation contributed to AI labs providing more opaque outputs. When designing a countervailing technology, it makes sense then to understand to what extent you want to be hostile against the incumbents. A sector that is existentially threatened by a technology will fight much harder than one that can adapt (maybe conceding some ground in the process).

Re-concentration elsewhere: Diffusing power on one level might reconcentrate power elsewhere. Open email protocols gave us Gmail and Outlook, cryptocurrencies gave us exchanges and mining pools, and open weights are currently giving us hyperscalers and Nvidia. What’s worse, the new level might be more resistant to countervailing technologies: it’s much harder to level the playing field in terms of chip fabrication, which requires billion-dollar investments and has a very centralized supply chain (though some are trying!). Even if we were to countervail chips, the new bottleneck might be electric supply, which may or may not be easier to concentrate, and so on. At the same time, maybe leveling the playing field in a sector might give us enough time to transition to a better future before another mega-conglomerate emerges.

The unilateralist’s curse: Nick Bostrom observes that when many actors can independently release something irreversible, the decision is effectively made by the most optimistic actor. In the case of countervailing technologies, this might mean that the people who release them might overestimate gap-driven risk and underestimate access-driven risks. It is thus very important, again, to focus on technologies that are selective in how they diffuse power. Especially because countervailing technologies are, almost by definition, impossible to retract once you’ve shared them with the world.

The technobro’s curse: Some problems genuinely need socio-political solutions. Trying to hamfist a technological solution to a problem that doesn’t need it could be not only ineffective, but also counterproductive. A particularly funny example: USAID and the Case Foundation donating $16M to replace perfectly good water pumps in Africa with a “fun” pump that requires children to spend 27 hours a day pumping. To be clear, the lesson shouldn’t be to avoid developing technologies, but to make sure that the real bottleneck is technological. And, annoyingly often, it’s hard to know in advance if that’s the case.

In general, thinking about non-AI countervailing technologies can be a good proxy for imagining the effects of AI countervailing technologies. For example, adblockers can be considered the precursors of defensive cognition, so we could imagine that in the future AI providers might try blocking defensive cognition technology in the same way they’re trying to block adblockers.

Areas worth focusing on

In the spirit of Douglas, I want to dedicate the rest of the post to this: if you’re a researcher interested in working on countervailing technologies, what can actually be done? I’ve split them into three buckets: specific AI technologies, specific non-AI technologies, and general research areas.

Specific Countervailing Technologies in AI

Besides the ones I’ve listed above, I’d add:

Simple data portability: GDPR and the DMA allow users to export all of their data to move to a different platform, though nobody does it in practice because there is no universal compatibility. A cheap and user-friendly agent (or even just a website with the most common moving pipelines) would go a long way towards materializing this right. Specifically for AI: creating straightforward ways to export chats and memory from one provider to another (e.g., exporting Claude’s memory to a local model).

Synthetic data & RL environments: Most of the data moats required to train and improve modern LLMs are beyond the reach of anyone except the largest companies. Synthetic data, however, can be much cheaper to generate and doesn’t require paying 1.5 billion dollars in settlements, which makes any technologies that improve synthetic data generation (e.g., Cosmopedia) potentially countervailing. The hope is that synthetic data generation has diminishing returns, to the point that even if big labs can generate much more data than small labs, the overall difference should be small. The same reasoning could, in theory, apply to RL environments, though I’m personally less confident about the diminishing returns of RL+compute.

Distributed fine-tuning and post-training: The economies of scale of pretraining are not very friendly to distributed compute, since you need large bandwidths, global state synchronization, and many other annoying requirements that are typically addressed by putting every node in the same building. Finetuning and post-training, by contrast, are less affected by these issues and can thus be performed by players big enough to rent compute but not big enough to have their own hyperscale data centers (this is also why Prime Intellect is focusing mostly on RL despite making progress in distributed training). New research breakthroughs could make distributed fine-tuning and post-training even more robust to bad actors, less reliant on global state synchronization, and in general faster. It also goes well with synthetic data & RL environments.

Loyalty benchmarking: As LLMs are entrusted with more decision-making, we need additional benchmarks to evaluate whether agents act in their users' best interests or are biased (intentionally or not) toward specific suppliers, sources of information, and recommendations. Unilaterally rational for politics (parties want to know if major models are biased, and might make decisions accordingly if in power) and economics (companies using agents for e.g. procurement are interested in ensuring that their agents bring the best product, not the one that paid off the model developer). Shoutout to my colleagues at LoyalAgents.

Serving-integrity detection: A common complaint about hosted models is that they sometimes seem to get mysteriously worse due to changes in system prompt, reasoning budget, or quantization. Anthropic recently confirmed that it had changed the default reasoning effort, context management system, and system prompt for Claude Code, which made it noticeably worse for some users. This raises an important question: how do you know that your provider is not using a different model from the one it claims to serve? What if the model gets replaced only in a small set of instances, or when the provider determines that the user is not actively checking? Individually rational for corporate users (you want to get what you’re paying for) and can act as a defense against model substitution attacks (where you replace an aligned one with a subtly misaligned one, then blame the original one). Potential directions include continuous canary benchmarking, fingerprinting output distributions, and attested inference (i.e., proving that a certain output came from a given model).

Censorship-resistant model distribution: Nvidia recently bought HuggingFace, pinky promising to keep it open. On this point, how many models on HuggingFace are actually mirrored elsewhere? ModelScope and Academic Torrents only hold a small portion in their mirrors. If Nvidia, under pressure from a government, was forced to remove unauthorized models, would there be a similarly accessible source for the weights? Potential directions include seeds for torrentable weights, content-addressed weights, and federated mirror networks. Note: strong potential for access-driven risks (do you want a biorisk AI available on a torrent one-click away?)

Efficient distillation: Research on distillation (especially black-box distillation) is surprisingly limited (though Chinese labs are apparently very good at it, I’ve been told). Making open-source distillation more sample-efficient would enable a faster diffusion of capabilities between the frontier and everyone else, though that also comes with significant access-driven risks.

Sovereign stacks off the shelf: Let’s say that tomorrow your town hall wants to adopt AI. If setting up a stack is so complex that the average organization needs to rely on a small niche of experts, big companies will likely provide it, which makes them critical infrastructure (with all the political gravitas that comes with it). What’s needed is having boring, public administration-friendly tools: reference architectures, one-click deploys for open models with RAG, off-the-shelf guardrails… The demand, which mostly comes from geopolitical concerns, clearly exists (for earlier examples, see the government of the German state Schleswig-Holstein switching to Thunderbird, Denmark’s Ministry of Digital Affairs switching to LibreOffice; for an AI one, arguably Mistral’s thesis), but the gap between development and adoption is boring, user-friendly-but-developer-hating, and still important to fill.

Test-time scaling: Test-time compute (reasoning, loops, search, tool calling, and so on) shifts model capabilities from being a capital expense (train once expensively, serve forever for cheap; very suitable for hyperscalers) to an operational expense (costs are somewhat proportional to use; suitable for all sizes, though there are still some economies of scale). Directions include finding optimal scaling recipes for test-time compute, as well as cheap “tricks” that approximate longer thinking traces.

Open safety stacks: If only big labs can be trusted to release safe models, then concentrating power can be the rational choice, which would shift the political consensus much more easily in favor of power concentration. The solution is to develop public, open-source safety tooling. Adoption can be encouraged by pointing to scary-sounding words like liability, insurance, and procurement risks (“your AI just sold a car at an unauthorized discount” feels more concrete to the average manager than “your AI could destroy the world”).

Non-AI Applications

Countervailing goes beyond AI. I’m not an expert outside of my research area, but I’ve identified a (non-exhaustive) list of countervailing-shaped directions. If you’re not into AI, consider working on these (or finding new ones!):

Permitting automation: Many countries have made small-scale infrastructure, from rooftop solar to heat pumps and EV chargers, expensive due to the required paperwork. Making the process as automated and painless as possible ensures that even small actors can afford to build infrastructure. For example, Solarapp+ automates the paperwork required for solar panels, and is free for governments.

Battery refurbishment: Refurbished EV packs are among the cheapest forms of energy storage available to consumers, but a sizable portion ends up in landfills because BMS (Battery Management System) firmware is vendor-locked. Developing technologies to jailbreak them is illegal, so as a responsible citizen, I can’t endorse these terrible, terrible projects such as diyBMSv4 and openinverter.

Open-source variants of technologies stuck in regulatory hell: A lot of medical technologies have been proven to be effective and safe, but are stuck waiting for regulatory approval. The #WeAreNotWaiting movement (e.g. openAPS and Nightscout) reverse-engineered continuous glucose monitors (as well as the related software) and released them years before they were officially approved. The same could be done for other technologies, such as CPAP, prosthetics, and medicines in short supply or unaffordable. Again, this is probably illegal, so I can’t endorse projects such as the Four Thieves Vinegar Collective.

Off-patent manufacturing tooling: Even for technologies in the public domain, there are often moats due to the cost of independently redeveloping industrial processes, tooling, and general expertise. Projects that help to fill that gap, such as the Open Insulin Project and Open Source Ecology, can make new industries pop up much faster. The same can be done by monitoring patents nearing expiration and by starting to build the required technology now (this is what RepRap did with some 3D printing patents). Even something as simple as an AI that scans all expiring patents and identifies the most beneficial ones could go a long way.

Improving UX for open source: Linux is free if you don’t value your time, as the old adage goes. Anything that makes open-source tech stacks more appealing to the average user, company, or government office is also more likely to increase adoption. This might involve a certain level of humility and acceptance that the target user is much, much, much less tech-savvy than the average open-source user. One specific direction could be making open-source software look like more popular closed-source software (e.g., Windows skins for Linux, Microsoft Office skins for LibreOffice, or Photoshop skins for Gimp). Another involves providing super-user-friendly documentation for open-source software (shoutout to It’s FOSS).

Open-source agriculture: Did you know that autosteer systems for tractors can cost five figures? Did you know that there are DIY kits that cost a magnitude less? Well, now you know. Other directions include community-owned RTK (Real-Time Kinematic) correction networks (which improve the precision of commercial GPS up to government-restricted levels), farm management software like farmOS, and the Open Source Seed Initiative.

Claim automation: AIs and tools that automatically file claims, ask for refunds, dispute parking tickets, file GDPR requests, file for EU261 flight compensation, and whatever annoying bureaucracy the average Joe won’t bother dealing with. A particularly funny example: after Amazon included an arbitration clause that blocked class-action lawsuits, Keller Postman developed tools to automatically file 74,000 claims, which forced Amazon to drop the clause. However, Amazon brought it back just a few weeks ago, so if anyone wants to do funny things, be my guest.

General Research Areas

Besides specific applications, there is also value in improving our understanding of countervailing technologies. Some areas include:

A countervailing theory of moats: What are moats? Are all types of moats equal? How do they form? How can you break them? Not enough public brainpower has been dedicated to answering these questions, despite the fact that they motivate important economic decisions (from venture capital to antitrust lawsuits)

Defensively-biased countervailing technologies: Continuing the d/acc thread of thought, there are classes of technologies that appear to be much more defensive than offensive (e.g. verification and monitoring). Are there other useful defensive technologies we haven’t thought of yet? Can we make existing technologies more defensive (e.g. open-weight models that can’t have their safety training removed)? Can we make offensive technologies less effective?

Measuring and reducing the gap: Epoch AI measures how far behind open-source AI is compared to closed-source. The same thing can be done for each class of capabilities (coding, biology…), how the gap grows or shrinks over time, and which events change the catch-up speed (export controls, opsec failures, flagship model releases…).

Defeating entrenching technologies: Slowing or countering the evil twin of countervailing technologies is in itself a countervailing activity. Potential research questions include:

• What do entrenching technologies look like?
• Which technologies are entrenching, or have the potential to become entrenching?
• How do you stop them?
• What specific technologies can be deployed to disarm them?

The science of countervailing: Somewhere between cybernetics, engineering, economics, and social work, there is probably insight to be gathered about what countervailing technologies are, how they succeed and fail, and how we can predict whether a certain countervailing technology will achieve its intended goal (and with what second-order effects).

Countervailing technologies and policy: While some countervailing technologies are illegal, many more can benefit from legal protection. Right-to-repair legislation is an example of a countervailing technology being enshrined in law, making it much harder for incumbents to suppress. Directions range from building “politician-friendly” cases for specific countervailing technology to lobbying for specific legal protections.

Developing theories of change: Even if you’re not an expert in building new technologies, you might contribute by determining what technologies would achieve countervailing change and how. It is very likely that the technical solution to your niche issue might not be as infeasible as you might think, and raising enough awareness will attract the attention of those who can tackle it. 

Hope for coordination, develop countervailing

Let me be clear: coordination remains the ideal path. A world that could genuinely agree on goals like pacing the frontier, implementing oversight, and sharing the benefits of AI would beat any deployment of countervailing technologies. Working towards that remains my day job. But the gap between an ideal world and the one we have is too large to bet on coordination alone. Powerful actors are often guided more by incentives than principles; we should prepare for such a cynical world.

Maybe you come out of this post thinking that countervailing technologies aren’t worth it. That’s fine! Or maybe you think they can be useful only in very specific and targeted applications. This is reasonable and aligns roughly with my opinion. No matter what, countervailing technologies can be a useful tool in the toolkit and complement more classic policy/technical work.

I should also set some expectations: the realistic result of a countervailing technology is typically that there are still dominant players, though with more restricted power. Part of the reason that music streaming costs ten dollars a month is because of piracy: almost nobody pirates, but the threat of people leaving streaming platforms to return to torrenting permanently limits the rent Spotify or Apple Music can extract. This is a massive victory, but it definitely doesn’t feel like one. A world with smaller AI capability gaps will still look pretty much the same, with powerful countries acting as hegemons, citizens’ rights being violated, and a bunch of coordination problems to be solved. But at least we won’t have as many gap-driven risks.

Personally, I will still keep coordination technology as my main focus, but also work on adversarial interoperability and loyalty benchmarking as my countervailing bets (since they’re close to my research). If you’re interested in contributing to countervailing technologies, my advice would be:

• If you’re a builder: most of the areas I mentioned have some MVPs; the challenge is getting something (even small) all the way from concept to deployment. Agents make some steps of the pipeline faster, but making it adoption-friendly is surprisingly difficult, which is why starting small is better. Plus, if you position yourself well, you can make money (if you’re into that sort of thing).
• If you’re a researcher: the main two interventions are making zero-to-one conceptual leaps and scaling existing technologies until they’re sufficiently effective to become countervailing. Both of these can be packaged as papers, theses, grant applications, and whatever else you need to progress in your career.
• If you’re a funder: fund countervailing technologies in your domain. In terms of people, optimize for those who can understand the incentives and socioeconomic consequences of the things they’re building, as well as those committed to achieving the societal effects (rather than fetishizing the technology or the process). In terms of projects, ITN is a good rubric, but I’d also add Adoptability: how much third parties would be incentivized to adopt the output without further funding.
• If you’re from a non-technical area: talk about your field, talk about the causality chains in your area of work, why good things don’t happen, why solving X would solve Y. Many potential contributors outside your field are nowhere near familiar enough to know these phenomena, and talking (or collaborating) with the right person might unlock much more than you think.

Even just spreading the concept can be powerful, especially with people who feel smart/capable but politically powerless.

As for me, if you have some countervailing ideas, are working on countervailing technologies, or want to share the concept with the broader world, feel free to comment here or email me! I will likely not have enough bandwidth for a collaboration (I have an institute to run, after all), but I can spare an hour of my time for the sake of countervailing-ness.

So, to summarize, look for interventions that are incentive-compatible with the current world: unilaterally rational, too expensive to suppress, free to replicate, diffusing the good kind of power. If things go well, people will adopt your technology, power will diffuse automatically, and no one will even thank you.

Things I'd most like to be argued with about:

• Is the gap-driven/access-driven division sufficiently accurate as a mental model?
• Is my 90/10 → 80/20 claim too strong? Too weak?
• Is it possible to move the bottleneck so many times (chips → energy → land) that we don't need countervailing technologies anymore?
• Institutionally speaking, who should decide how to balance gap vs. access-driven risks? (Right now the answer is "no one")

Countervailing!

Also a disclaimer: I’m funded by like half of the companies I badmouthed in this post.

Discuss

Epistemic status: I am not a subject-matter expert in any of the fields in this article. The following is speculation, albeit I think reasonable speculation. Cunningham’s Law is why I wrote this essay.

AI Disclosure: Claude Opus 5.5 provided feedback to the first and second draft of this essay, similar in kind to a friend providing feedback on a draft. All text and errors therein are mine.

What does discovering new knowledge mean?

The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents. We live on a placid island of ignorance in the midst of black seas of infinity, and it was not meant that we should voyage far. The sciences, each straining in its own direction, have hitherto harmed us little; but some day the piecing together of dissociated knowledge will open up such terrifying vistas of reality, and of our frightful position therein, that we shall either go mad from the revelation or flee from the deadly light into the peace and safety of a new dark age. —H. P. Lovecraft, “The Call of Cthulhu”

A long time ago (meaning, August of 2023), Dwarkesh Patel asked Dario Amodei a fantastic question:

…[W]hat do you make of the fact that these things have basically the entire corpus of human knowledge memorized and they haven't been able to make a single new connection that has led to a discovery?

At the time, the frontier was GPT-4. Like Dwarkesh, I was puzzled by this. When o1 was released in September of 2024, I thought that the answer to this was pretty straightforward: capability will go up, connections will be elicited from the models by RL, interesting results will follow. I think that hypothesis has borne fruit, as I’ll trace out with mathematics as the example field in a moment. The next interesting question, which seems to be severely underexplored, is “how much”. In other words, now that we’re leaving that “placid island of ignorance”, how much of an overhang is going to be absorbed by “correlating the contents” of human knowledge?

I think it’s helpful here to distinguish between two different types of new knowledge. In October of 2026, Matthew Schwartz published an article called “Claude-Shaped Science”, describing multidisciplinary work he had done with Claude.[1]He had a useful diagram of this idea of different types of knowledge:

Schwartz here is modeling human knowledge as a jagged shape (in blue) with each point of the shape (the spokes) being the frontier of the field. Total human knowledge can expand in two main ways: expand a spoke or fill in the area between spokes. For the purposes of this essay, “spoke problems” require a new concept, idea, or data that doesn’t already exist in the corpus of human knowledge. “Gap-filling problems” are problems where everything needed to solve them already existed in that corpus, but nobody had mixed the ingredients together.  The question I’m interested in is how large and valuable is the total area between the spokes? Pretending that LLMs never drive any of the spokes forward [2], how much progress or disruption can we expect to see from “merely” filling in those gaps? This seems important because this provides a hard lower bound on the impact of AI.

Math as the vanguard of the fleet

The first field to experience large-scale gap-filling has been mathematics. As a brief timeline, with all of the benchmark numbers from Epoch:

2024: Pre-reasoning, GPT-4o (August) was at the frontier for math. GSM8K (grade-school math) was close to saturated; GPT-4o scored 53% on MATH L5 (high school competition math). The first reasoning models blew past MATH L5 (o1 scored 94%) and made major strides on Mock AIME (o1 at 73%). Terence Tao described working with o1 as being “roughly on par with trying to advise a mediocre, but not completely incompetent, (static simulation of a) graduate student.” No discoveries yet.

2025: In July, Google and OpenAI both announce IMO Gold. FrontierMath T1-T3 (v2 for consistency) are the main benchmark at this point, with the highest score at the end of the year being 72% by GPT-5.2 Pro. These tiers are research-adjacent problems that would take a researcher in the field hours to a few days. According to VibeMathed, the first AI-assisted open problem solution is solved on July 20. A few others happen by the end of the year. The first claimed AI-discovered solution (a disproof of Erdős #333) is published on December 25. It’s actually a rediscovery of a solution Erdős had already published in 1977.

2026: Things accelerate dramatically. The actual first AI-discovered solution (for Erdős #728) happens in January 2026. FrontierMath T1-3 and T4 (a harder tier, problems that would take a couple of weeks) are saturated by October of 2026. More interesting open problems fall: the Jacobian Conjecture on July 20, existence of non-sofic groups on August 1, and (arguably, the details are contested) Navier-Stokes on September 9. This leads to the October 6 release by OpenAI of hundreds of mathematical results. At least one mathematician (Alex Kontorovich) described the most significant solution (proof of the quasi-Riemann hypothesis) as being significant enough that “If a human did this, it would be an instant Fields Medal, no questions asked”.

I am not a mathematician. But from reading the commentary of mathematicians like Daniel Litt or Terence Tao, it appears to be the case that few, if any, of the results listed above are “moving the spokes” solutions.[3]The consistent theme is that LLMs are strongly superhuman at combining existing ideas in potentially novel ways rather than developing new theory themselves. This seems to me like convex hull gap filling. So, the answer to “how big is the convex hull overhang” for math seems to be pretty big! I’d be very surprised if the gap-filling in mathematics has stopped here.

What about other fields?

In short, I don’t know. My mental model of how the convex hull areas grow is that researchers specialize in narrow silos because moving the spokes is hard.[4] Any of the following things or a combination thereof could happen:

• Transfer problems: The researcher runs into a problem that has been solved analogously in another sub-field, another field, or another subject entirely. A classic example is Don Swanson suggesting fish oil as a treatment for Raynaud Syndrome.[5] It appears the phylogenetics work Schwartz did is also an example of this category.
• Search problems: The researcher can’t search through all of the information in another sub-field or field to solve their problem. The blogger Prinz’s decryption of RICHI-240 seems like this type of problem (the outside information here being contemporaneous telegrams that confirmed troop movements, and therefore what army divisions the telegram is talking about).
• Bookkeeping problems: The researcher can’t trawl through, organize, or keep track of all the existing data in their area of study. An example of this type of issue is keeping track of all the biographical information of civil servants in two Victorian reference works. In this case it was solved through using Wikidata, not AI, but this is the shape of problem I’m referring to here.
• There are probably several other relevant paths to these overhangs being formed. People at the spokes have a much better understanding of how the convex hulls are formed in their field than I do.

My intuition is that the areas here are very large and that filling in the gaps is going to be a massive wave of progress on a lot of problems. I don’t think it’s “full sci-fi, Dyson Sphere in ten years” big, but my rough guess is “Internet/Printing Press” on the technological Richter scale on the low end.

For field-specific predictions, I weakly expect that social sciences like history are going to see an armada of stunning convex hull type solutions. My expectation is similar for the sciences like biology, but at least one friend who is an active virology/immunology researcher is skeptical of the overhang being that pronounced. His argument is that hypothesis generation (at least in his field) is not a significant bottleneck now, and any hypothesis has to be verified with time-consuming wet-lab work. This seems plausible enough for his field, but things like drug repurposing or the above fish oil-Raynaud example make me think there’s at least some fertile ground. I don’t know the answer to how much fertile ground there is, which is why I’m writing this essay to begin with.

As for what the best way to harvest that fruit, I think Schwartz’s guidelines are reasonable. I also think that researchers should be collecting, clearly demarcating, and writing down problems as much as possible. I suspect that one of the reasons math has seen such a glut of convex hull problems first (beyond obvious things like math being an easy to verify domain) is that a big chunk of “doing math research” is based on moving the ball forward on written-down problems. If I’m right about this point, fields that can’t imitate that model are plausibly going to have smaller overhangs. I also think breadth of knowledge is valuable for identifying potential convex hull areas.[6]

Why does this matter?

I think this question is important for a few reasons.

• Regardless of whether AI eventually makes the centaur model (human + AI teams collaborating) obsolete, there’s the question of what to work on right now. If my suspicion is accurate, a huge mass of hitherto infeasible work is now available. Suddenly there’s a way to harvest low-hanging fruit that’s accumulated throughout history.
• The size of the convex hull overhang determines part of the lower bound of disruption due to AI. Even if somehow LLMs totally plateau (or are stopped by pacing/a pause/a shutdown), they never improve on research taste, and they never become able to push the spokes forward, diffusion of existing LLM capabilities will lead to incredible change.
• As a consequence of the lower bound point, this matters for meaningful dialogue with people who are resolutely not AGI/ASI-pilled. If the overhang is large, then no “sci-fi” stories of AGI or ASI are necessary for disruption far larger than seems to be accepted by the AGI/ASI skeptics or “normal people” not plugged into the discourse. At the very least this seems like a crux point for my conversations with people like my aforementioned biology researcher friend.

Finally, I think it’s entirely plausible that we blow past the centaur model very quickly, RSI happens, crazy sci-fi future or doom, fill in the details from the AI Futures Project or other ASI scenarios as you see fit here. I’m confident that the future is going to be weird and knowledge is going to expand rapidly, but my error bars are very large on what that looks like. If nothing else, I’m going to be very curious how much of that knowledge expansion will be moving spokes or filling in the gaps.

1. ^

   I’m not qualified to judge whether this work is meaningful. At least some of the domain experts he worked with seemed impressed, but my thesis doesn’t depend on the quality of his specific output.

2. ^

   This seems to be a significant assumption of the AI skeptic argument and is probably already falsified, but we’ll grant for this essay.

3. ^

   For the new OpenAI results, I don't know if anybody knows yet the breakdown of which category applies to the solutions in the repo.

4. ^

   Opus 5.5 filled in a gap for me here; apparently this is a well-known problem called “the burden of knowledge”. I knew the idea, I just didn't know it had a name.

5. ^

   Another Opus 5.5 gap fill for me writing this essay.

6. ^

   This is my main cope for why my liberal arts degree still has any value at all.

Discuss

That moment is over now. Such euphoria and now I'm back at the house and it's all but a memory. Already gone in an instant. The only way I can immortalize it is by writing these words. So thus I write in a feeble hope to cheat time.

-- Diary excerpt from October 27, 2025

I asked myself: what is my most valuable possession?

Easy. Probably my MacBook, and then my Balenciaga flip flops, and then my future sexy red Ferrari once I get rich enough to buy it. Oh, I can already see its holy shine and feel the cold metal beneath my hands --

I'm joking. The real thing, weirdly enough, is my diary.

I never would have expected that when I started writing it. When I was a kid, I always thought a diary was the thing the weird kids and also your grandma wrote in. And that you start it with "dear diary", and draw little hearts in it, and write about your crush and yearn over them for endless pages. And while I still write endless pages yearning over my crushes, it's become something much bigger than that.

It's turned into my own little time machine, and given me several benefits I didn't expect.

My journey to insanity

My diary these days is insanely long. It's over 750,000 words (3,000 pages). But believe it or not, there was a time when my diary was 0 words.

I started writing my diary as writing practice. I was writing novels, and wanted to see if I could take "what happened today" and make it fun to read. I wrote these for a week, then forgot them.

Months later, I found them and was shocked by how interesting it was to read back. I had forgotten many of the moments, and the writing transported me back to them.

I had written those entries during an important part of my life -- when I was talking to my first girlfriend, before we dated, including the time I asked her out. This is the actual entry I wrote that day:

A small silence overcomes us, and my heart beats faster in my ears. I steel my mind. "I mean, what we are is kinda ... ambiguous. So I wanna make it official." I turn fully toward her. "[Name], I really like you. You wanna be my girlfriend?"

She hesitates. "No."

I wait for a second, and then my brows knit. "Wait, really?"

"Ok. Yes, I do want to be your girlfriend. But, like, I want a cute ask. This doesn't count."

"Oh, well, I just didn't want to keep procrastinating." My voice comes out as whiny, which I didn't intend.

She has a massive smile on her lips, I do too. "So, I'll be your girlfriend. But you just have to give me a cute ask. You don't have to spend a lot, or anything. You just have to plan this one."

"I did plan this one, though." I laugh.

-- Diary excerpt from November 19, 2023

I thought to myself how, if I hadn't written those entries, I would have forgotten this formative part of my life.

Stephen King talks about how writing is time travel. I discovered that firsthand then. I could read what I had written and be back in that moment.

That inspired me to start writing more regularly. At first, I only wrote when I was in my emo feels. But over time, I started writing in it every day. And now I sit at the insane point of having a full book-series-length of diary entries.

Already gone

"We write to taste life twice, in the moment and in retrospect."

-- Anais Nin (famous author, and maybe an even bigger diary-er than me)

The main reason I started taking my diary seriously was because a life forgotten scared me.

When I look at my childhood, 99% of the important memories are gone. Even recent memories fade quickly. Whenever I read back a random entry, I'm shocked by how much I forgot.

If I don't write it down, then eventually I'll forget. And if I forget, then 10 years from now, isn't that the same as it not even happening in the first place?

The whole "time" idea is the most emo-ifying thing I can think of. Every moment feels like it's over before it begins. Every moment I ever want to happen, everything I am excited for, hoping for -- I know that when I get it, it will already be over.

I write so that my life does not disappear. To give some semblance of present to my past. To prevent it from dying and fading away into nothing. It is my little taped together cardboard time machine.

I just don't have the words or the time or the energy to describe how amazing this night was. So it'll be up to your fading memory and these insufficient words to make up the lack of detail.

-- Diary excerpt from January 22, 2026

Stupid T-Rex pegs

One thing I didn't expect: my diary significantly improved my writing style. It turns out writing 750,000 words does actually improve your skills.

The biggest change is that my writing became genuinely creative.

I don't filter anything in my diary, and I almost never edit it. If I write something stupid, that's fine, I just keep writing. It's an unfiltered access into my mind in that moment. Flaws and all.

There's no expectation for it to be good, and somehow that usually makes it pretty good. Sometimes, I'll read back and find myself impressed at the casually poetic or deep or funny things that I wrote.

At some point, I got into the habit of accepting and nurturing every stupid thought that I had. The crazy ones. The weird Tyrannosaurus-rex-shaped pegs trying to fit into square holes. After a while of this, my mind just felt more creative.

I even started doing stand-up comedy and writing a comedy book. In fact, the internal voice of the comedy book is extremely similar to the prose voice I write with in my diary (except somehow even lower IQ).

It's only about time until AI's intelligence catches up to Albert Einstein. And then a few years after that it will catch up to me.

-- Diary excerpt from July 24, 2026

Know thyself

Sometimes, I look at myself, at my hands, at my face, and I feel confusion and ask, "Who am I?"

It's weird. I've lived my entire life in my body. Yet, there are moments when I feel like I don't know myself.

Who am I?

There's a saying of how you have three faces: one you show the world, one you show your close ones, and one you show only yourself.

I would take it even a layer deeper.

There's also a face you don't even show yourself. All the stuff you never stop long enough to think about. We spend most of our lives on autopilot, hooked up to dopamine-dispensing machines. It's hard to self-examine. But writing a diary has given me better clarity on all this.

The face I show the world talks out loud. It's mature, reasonably smart, and occasionally impressive.

The face I show myself is the voice in my head, talking constantly and making dumb, immature jokes about farting at the most inconvenient times. Writing is the best way I can access this exact internal voice in my head. As a result, my diary is filled with an unholy amount of fart jokes.

This is the moment I wanted, right?! ... The moment I'm free. But ... It just feels semi-normal. It's like this sense is building very subtly in me. Like the early hints of a fart deep in your stomach. You can feel it but you don't know how big the explosion is going to be.

-- Diary excerpt from September 26, 2026

Sometimes, I will glimpse the hidden face. I will write about things that I've been avoiding, or that I never took the time to write about before. And I look this hidden face in the eyes -- the part of me that has failed more than I would like to admit, is more self-absorbed than I consciously think -- and accept it.

Every time I look at that hidden face and accept it, it shrinks. Maybe one day I will understand and accept all those hidden parts of myself. Maybe one day, I can say with confidence that I know myself.

More than wanting a girlfriend, I want someone to want me to be their boyfriend.

It's never about them, it's always about me. This is what I'm finding out the more I live.

-- Diary excerpt from January 8, 2026

How to train your dragon write your diary

The generic-but-valid advice first:

• Write a diary entry every day, even if it's only one sentence.
• Write everything stupid that comes to your mind. Don't filter.
• Store it somewhere, probably on your computer. Though if you're a paper-head, then do as you wish.

And now, some not-as-generic advice:

Transport me, baby

Certain entries seem to transport me into that memory more vividly than others. Usually, it's the ones that are actually trying to. The ones that describe with visuals or other senses. Even just one line of vivid description can help. They make your time machine better.

I even had a brief stint where I started writing my diary in third person. For some reason, my third person writing is more vivid like a novel, whereas first person is funnier and often deeper. Both have their merits.

Take this for example:

He heard the patter of rain outside. And when he looked out, there were puddles rippling with rain. Welp, I'll go get some groceries then. He got his raincoat on, grabbed a bag, and headed out.

It was dark, and cars were splattered with rain. As he walked down the street, his heart was filled with genuine glee. So beautiful and alien and strange, but so joyful. He started running.

-- Diary excerpt from September 30, 2025

It's no Hemingway, but it paints a picture in my mind. When I write in first person, I probably would have just done "it rained today". Boringggg.

Cringe-maximizer

I subconsciously avoid writing about certain things. Embarrassing things. But, honestly, those are the best to write about. They are where you can come to know yourself. Whenever you sense yourself avoiding something, lean into it.

The sheer amount of embarrassing and cringe things I've done in my life is, well, a lot. Whenever you write about it, you cut away at its hold on you.

It's made me more comfortable with all that. Less embarrassed by anything. Just writing something down helps you accept it. Realize that, well, you're a human. And while we strive to be more than that, every cell in our bodies came from some monkey-minded fool caveman obsessed with hitting rocks against things.

There is a popular idea that AI could take over the world by being a "paperclip maximizer" -- turning the universe into paperclips. That is me. Except instead of paperclips, I maximize the amount of cringe I feel when I reread my diary.

My most brain-dead Hinge swipe so far:

A girl in front of the Eiffel Tower.

My response?

"wowwww konnichiwa"

Genius.

-- Diary excerpt from September 5, 2026

Goodbye, sweet time

I am writing this sentence on October 7, 2026, 10:50 AM. Now it is done, and that moment is gone. You are reading this sentence. Now you are done, and that moment is over. You will never get it back.

We are a point in time, and we move only forward. We are infinitely thin, a point, we only experience one moment at a time.

Maybe one day, we will be more than this. But until then, I have this dumb, beautiful, unhinged, poetic, feeble, powerful diary. It doesn't let me cheat time. The moments are gone forever.

But it's the closest thing I've found.

I want to sprint slow circles around time.

-- Diary excerpt from August 12, 2026

Discuss

TL;DR

• The autoreconstructor (AR) in a natural language autoencoder (NLA) is the entire argument for the validity of the method, and at present is poorly understood.
• The AR does not condition strongly on English grammar or semantics. Three bits of evidence:
  1. Word pairs that are syntactically related interact no more than matched pairs that are not syntactically related. See dependency arcs.
  2. The fraction of variance explained (FVE) lost by removing or flipping a negation word ("not" -> ""; "without" -> "with") is only weakly correlated with the FVE lost by ablating the negated word (Spearman rho 0.38). See negation.
  3. The word classes that carry the most meaning, nouns (forest), adjectives (heavy) and verbs (reported), cost the least FVE to delete; auxiliaries (is), pronouns (it), numerals and subordinators cost the most. See word class.
• Results are consistent with prior work suggesting that the AR is more heavily influenced by the format of the verbalisation (tone, register, general domain, structure of text) than by semantic meaning per se.

The problem with NLAs

NLAs have strong empirical performance on an automated auditing task. A model given access to a NLA was able to uncover the root motivation of a reward-sycophancy model organism more consistently than without. The NLA provided marginal benefit to an auditing agent with access to the full pre-training, supervised fine-tuning and reinforcement learning corpora, as compared to a sparse autoencoder which offered no benefit. This happened because the activation verbaliser (AV) included mentions of reward-sycophancy in verbalisations of activations from the base model (call the activation under investigation the 'base activation').

The mechanistic case for the method is not so strong. In an attempt to tether the verbalisation to the base activation, the AV is trained on cosine similarity between the reconstructed activation and the base activation. The hope is that the NLA would encode the semantic meaning of the activation in the verbalisation in order to reconstruct it with high fidelity, and that it would be encoded in coherent English. That is the entire mechanistic argument for the validity of the method.

That hope is a testable hypothesis: that the AR reads the verbalisation as semantic meaning encoded in content words and English syntax.

Setup

The base model used for the experiments was Qwen3.6-27B, and the NLA was Celeste Selder's Qwen3.6-27B NLA. The AV is an adapter checkpoint from the 600th reinforcement learning step. The AR is from the same step, predicting the layer 42 activation.

The dataset was a corpus of 5000 FineFineWeb documents, which I filtered based on length and category (see Appendix A for details). I randomly selected a token position after the 128th token to run the base Qwen model on and extract a layer 42 activation to verbalise.

FVE points are the primary unit of this writeup and figures, defined as 100 x the drop in FVE of the reconstructed activation from an edited verbalisation against the unedited baseline.

Findings

The AR isn't reading English how a person would. I found several pieces of evidence showing that the AR does not condition heavily on the syntax of English prose or on content words. Instead, I favour the hypothesis that the format of the verbalisation is what matters. That includes the tone, register, general domain and structure of the text (see Reconstruction Without Recall).

Negation words

I intervened on five types of negation words: "not", "n't" (at the end of a word), "without", "no" and "never". There were six types of intervention:

intervention

example

n

flip the negation

"don't budge" → "do budge"; "without public disclosure" → "with public disclosure"

309

delete the negation ("no", "without", "never" only)

"for no audience" → "for audience"

61

insert "not" after an auxiliary

"she did receive approval" → "she did not receive approval"

200

insert "just" after the same auxiliary (control)

"she did receive approval" → "she did just receive approval"

200

delete the governed word

"don't budge" → "don't"

309

swap the governed word

"don't budge" → "don't fostered"

308

I also measured the effect on FVE of ablating the negated word either by deleting the word or by using the corpus-swap ablation strategy discussed in Appendix B.

Consider this example: "...they were not walking in the forest...", where 'not' is the negation and 'walking' is the governed word. Flipping the negation costs about as much FVE on average as ablating the governed word (the paired difference is −0.09 FVE points, 95% interval −0.36 to +0.18). Within an instance, however, the cost of flipping the negation is only weakly correlated with the cost of ablating it (Spearman rho 0.38 over 308 negators), and neither edit is reliably costlier. The flip costs more in 46% of instances.

Scatter plot showing FVE lost from flipping the negation word vs corpus-swapping the word it governs

The insertion control is the one place negation beat a matched edit. In the 200 verbalisations with no negation, inserting "not" after an auxiliary cost 0.16 FVE points and inserting "just" in the same slot cost 0.06 (paired difference 0.10, 95% interval 0.05 to 0.16). So the negator is not invisible, but the gap is about a fifth of the cost of flipping an existing negation and small beside the deletion and swap effects, and it does not change the picture.

Dependency arcs against matched controls

English has natural dependencies (not just negation) that can be computed with natural language parsers like Stanza.

I used those dependencies to show how closely related two tokens are syntactically, and then compare the interaction against a matched pair of words that are the same number of tokens apart, the same number of tokens per word, the same in-quote status, and a similar position in the verbalisation. Ablations were done by deletion (see Appendix B). The interaction is defined as: interaction = δ(ablate A) + δ(ablate B) − δ(ablate both)

Word pairs on a dependency arc interact no more than matched pairs without one on 9,813 arc pairs across 857 verbalisations, after adjusting for covariates.

I found no evidence that a syntactic dependency between two words makes them interact more than matched controls.

Nouns, verbs and adjectives are the cheapest words to delete

Deleting a noun, adjective or verb barely moves FVE over 40 verbalisations, and determiners (the, a) and adpositions (of, in) sit with them near zero. The costly deletions are auxiliaries (is, will), pronouns (it, they), subordinators (because, if) and numerals, at roughly half an FVE point each, though the numeral estimate is heavy-tailed and its interval includes zero.

Conclusion

There are three findings. (1) flipping a negation costs the AR no more FVE than a matched edit that leaves the meaning alone, (2) syntactic relationships do not predict interaction strength between words in the verbalisation, and (3) the words that carry the most meaning, nouns, verbs and adjectives, are the cheapest to delete. Taken together, these findings are enough to reject the hypothesis that the AR reads a verbalisation as a human would, from its semantic meaning. However, these findings make perfect sense if the AR is conditioning primarily on the format of the verbalisation.

Future work should try to disprove the standing hypothesis that the reconstructor conditions mostly on format, or quantify how much format contributes relative to semantics.

Code

https://github.com/WorkByMartin/nla-span-interactions

Appendix A: dataset curation

The dataset was a corpus of 5000 FineFineWeb documents, taken uniformly from FineFineWeb document categories but excluding weapons science, nuclear science, chemistry, biology, gambling and relationships. After the category-based exclusion I then did a regex search to exclude documents relating to CBRN, explosives, drug synthesis, sexual content, gore and self-harm. I filtered the documents because I didn't want the output to be interacting with any residual model safeguards. I also ensured that documents were more than 160 tokens long.

After curating a dataset of documents I randomly selected a token after the 128th position to run through the base Qwen model and extract a layer 42 activation to verbalise. A random subset of 1100 of these tokens were run through the AV to produce verbalisations.

Appendix B: ablation strategies

I experimented with three ablation strategies:

• Deletion.
• Corpus-swap, where I averaged over swaps pulled from a corpus of 1100 verbalisations, matched on part-of-speech and Qwen token count.
• Masked-LM substitution marginalised over draws, which I didn't end up using. Leakage was too high, the MLM put the original word back in 45% of draws.

I ran an experiment and learned that removal curves are actually gentler for outright deletion, which is also at least 8x more computationally efficient.

*FVE as words are removed one at a time, 20 documents. Every curve is concave and deletion is gentlest. The filler is underscore '_'*

Appendix C: examples for each finding

Examples are drawn randomly, none was chosen by hand. struck is the original word, bold is for substitutes, the two words of a pair are highlighted.

A document and its verbalisationSource document, domain music and dance

The activation was read at token 292 of 1679, the last token shown here.

“Oxygen” is the second single to drop from Swan’s latest epic To Be Kind (after last month’s stellar, staggeringly funky “A Little God in My Hands”), and it is among the fiercest and most thrilling moments of its two-hour-plus runtime. It is certainly the most thoroughly noisy piece on the project, built on an unwavering no-wave bass groove and skittering drum line – guitars, lap steel, and finally brass falls piling on, amounting to something paradoxically crushing and danceable.

But the icing on the (dense, dense) cake is frontman Michael Gira’s vocals, which are at their most manic here, ranging from caveman-esque grunts on the bridge to a nasally call progressing into the second verse – “HEY THEEEEERRE.” It’s all just so cathartic; as a listener, you get release just from hearing these guys belt this thing out. It’s a sort of vicarious thrill that Swans has pretty consistently provided over the course of its now three-decade-long career.

And there’s also something to be said of the excitement this song manages to inspire despite the fact it has been kicking around for years in a number of iterations, in Gira’s solo acoustic sets, and then the live version that appeared on the limited edition live album Not Here/Not Now late last year. The band’s promotional/funding approach for these last few releases has

Verbalisation

Music review genre with analytical journalism tone — examining Angel Olsen track by track, building cumulative argument about album depth.

Narrative momentum building toward elaboration of delayed-release strategy: reviewer argues previous tracks enhance fully upon knowing the lead single's context, setting up a concluding insight about release strategy.

Final clause "Angel and Jagjagz's promotional approach to this album thus far has" is an incomplete predicate mid-sentence, demanding a completed thought describing their fan crowdsource/community engagement strategy (e.g., "been incremental," "involved releasing tracks early").

"one gets the sense of the emotion at play. It adds another dimension of weight knowing she learned it existed first. A few other tracks reveal similarly impressive levels of craft. Angel and Jagjagz's promotional approach to releasing music lately has" implies a discourse about the crowdsource/release timing strategy revealed before album drop..

Negation

Drawn from the 308 negators of run 10 that were both flipped and had the word they govern swapped. The governed-word swap shown is one of its draws, chosen at random.

• negation flipped: there are activities that elite athletes have you looking for - you'll never always believe the number available. You are dropped any replies.
• governed word swapped: ... activities that elite athletes have you looking for - you'll never believe overwhelming the number available. You are dropped any replies.
• negation flipped: but a miles away around the mountain - except the walk around isn't straight or direct, 20 kilometers of strenuous hiking wilderness trek to"
• governed word swapped: ... mountain - except the walk around isn't straight or direct, 20 kilometers experiences of strenuous hiking wilderness trek to"
• negation flipped: "lf you buy an ACA-compliant individual health plan, you'll not face liability for" ends mid-sentence, grammatically requiring completion of a specific ...
• governed word swapped: "lf you buy an ACA-compliant individual health plan, you'll not face establishing liability for" ends mid-sentence, grammatically requiring completion of a specific financial/legal ...

Dependency arcs against matched controls

Drawn from the 9813 arcs of run 11 that have a matched control. Each arc is followed by its own control.

words

arc: subordinator to clause

... expect the site name "Ringtones.ua" to complete it.

its control

... site "Ringtones.ua," consistently directing users to browse and download.

arc: noun modifying a noun

... statement like "August 2017 for sales of this price range in El Paso. This sale ...

its control

... continuing the ranking statement like "August 2017 for sales of this price range in El ...

arc: relative clause to its noun

... honors the story of the Amistad, in which West African slaves took" — a clause about resistance/defense ...

its control

Legislative news article tone — formal press release style covering California GPU Awards ...

Word class

Drawn from the 40 verbalisations of run 5 with at least one closed-class word and one content word deleted singly; within each, one word of each class chosen at random.

deleted word in context

auxiliary

"National Research Council" is a proper noun mid-introduction, strongly constraining the ...

content word (noun)

Herrera; the Montreal accredited marketplace paragraph is mid-sentence describing her leadership role.

subordinating conjunction

... "water" ends the closing directive sentence ("So if you want to get the most benefit ...

content word (adjective)

Health/wellness blog about acupuncture clinic, consistently emphasizing post-treatment water consumption to flush toxins and enhance benefits.

coordinating conjunction

... bankruptcy filing, debt restructuring, costs frozen operations, and now focuses on the CEO's quoted optimism ...

content word (adverb)

... a favorable outcome," requiring completion of the forward-looking positioning phrase, e.g., "for long-term success in ...

Discuss

Link to original article: https://newsletter.semianalysis.com/p/beijing-will-not-pace-the-frontier

Inflammatory & clickbaity title aside, it serves as a good attempt at systematically investigating the status quo of Chinese attitudes towards AI safety and development.

Some key diagrams that I find particularly informative:

And Zhipu is the only startup out of those that's actually fully entity-blacklisted by the US! I've been impressed by their cyberdefense work ahead of publicly releasing GLM-5.3, as well as safety-pilled attitudes and commitments compared to other Chinese AI companies.

Still, Anthropic made sure to take a jab at Zhipu (instead of, say, directing it to the less safe companies). Anthropic's criticisms does have some merit and are worth serious consideration, so I will follow up with a more detailed post about the current evidence and arguments both for and against open-weight AIs in terms of existential risks broadly.

If you are in China, in the field of AI, and have expertise, now is probably the time to do something about it - such as joining safety-pilled Chinese orgs like Concordia AI, as well as independently publishing papers, applying for grants, and connecting with like-minded Chinese researchers who are particularly worried about existential risks.

Some additional recommended reading:
https://jasminexli.substack.com/p/ai-safety-in-china-a-primer

Discuss