Сборщик RSS-лент

I discovered it is surprisingly easy to accidentally indoctrinate your children. See, Twitter was aflutter with p(doom) renditions and I became particularly enamoured by this faux anime take by MakeVoid.

It’s cute! It’s peppy! It’s a direct causal link to me hearing my 5-year old sing “Orthogonality thesis blueeeeeee-ue-ue-ues”.

Now in my defense, I grew up agnostic. No Sunday hymns or evangelical pop for me. Just my sister awkwardly shushing me when I started cheerfully belting out “My milkshake brings all the boys to the yard” before anyone had explained low protein milkshake alternatives to me.

Now once my children were cheerfully reciting a first approximation of AGI Ruin, I did ask them what they thought the song was about. My 8 year old said it was about “ChatGPT please don’t eat us” and my 5 year old said it was about “AGI”. When I asked her what AGI actually is, she said it’s when the computer talks to you but you shouldn’t listen cause it might hurt you.

Oh.

Well.

I mean.

Kiiiiinda?

So, that was my crash course in the childhood memetic effects of catchy tunes. I briefly debated playing the competing songs of e/acc for them, but in the end decided I’d just be more mindful in the future and let them consume what they wished in the mean time.

Yet the power of song did not leave me, and I decided I wanted more.

I had already been playing around with Suno just before p(doom) dropped. The first time I fed it this tweet about the German wiki owner unknowingly holding back an AI swarm:

I couldn’t stop giggling at the dramatic rendition.

I tried making a few more songs, including one to help me write.

But when all is said and done I’m a music barbarian: In primary school music was the only class I managed to fail, which was honestly a bit of an achievement in itself. Yet I know what I like and now we have ways to create even the barbarisms that please one such as I.

So here we go. I want a song about myself and have it look cool. Oh mighty machine gods, will you grant me this gift?

Well, apparently yes, for $65 dollars a pop.

Which is super valid, but also I’d probably be tempted to iterate my life’s savings into the ground. So instead I decided to mush Astra and Fable together and see if they would make babies I mean art I mean my ego project.

I asked them both to look me up online, look through all their memories of me, and look through my social ranking data on Glosso (a social media website revolving around stats and dating).

People rank you on different traits based on their impressions so far, and then the ranking is weighted for how well they know you. I apparently have a fairly warm social demeanor. Hi <3

I asked them to invent song lyrics to describe me. Fable recited all my life events back to me as a grocery list and Astra saw straight into my soul. Suno then sang about said soul. And thus I found myself wistfully gazing at the p(doom) video and implored Opus 5.5 to please make a video for my soul song that would ascend to such heights. (if you are wondering why I keep asking new models, it’s because blinking is a known cause of new model releases and I couldn’t sustain a continuous stare through the entire project).

Still avoiding the 65-dollar-life’s-savings-trap, aka Fal AI, I asked Opus 5.5 to “just figure it out” and let it talk to ChatGPT on its own. I came back the next morning to phantom ghosts of smaller animation models now inhabiting my computer and the video results still being terrible.

I discovered that whatever character design you go with as a seed for the video will define the entire visual language and you better make sure your protagonist is your one true soul buddy come into pixelated existence or you won’t love them always and forever.

When I tried to storyboard these ladies to the same scene in the song I got three entirely different visual narratives.

Observe:

The Dgeneric Dreamy Drawing Damsel:

The Powerful Wrench Wench:

And What if Captain Ahab Was Light, Bright, And Entirely Unfit for Service?

So back to the drawing board looming over an AI to please get the details of my outfit right cause maps, wrenches, and going cross-eyed about a flower don’t resonate with my soul.

A few more iterations and we ended up with Hiking Gamer Girl who’s scarf should make her fly but this apparently dropped out of the context window ever after into the abyss where all 1s and 0s go to die:

And Magic Aesthetic Girl who likes warm things:

As we’ve now determined the full depth of my personality, we can finally start a storyboarding.

Actually I gave her necklaces for family members and bracelets for partners and friends. I wear these in real life.

After assuring Opus that yes, I really really like the aesthetic of that one p(doom) video. No don’t literally copy it. Yes use the character designs above. No don’t animate her as singing about herself. Yes she is allowed to open her mouth as long as she is not singing. No arms don’t move like that omg please stop.

I love you, Opus, but and also alignment is hard.

Joint alignment in humans, I mean.

Eventually we found the sweet spot at just rotating the images and karaoke highlighting the text.

So now I have an ego song and I’ve never been this excited to share something so deeply embarrassing.

What do you think?

Substack doesn’t allow video embeds and I didn't check if LW is better, so you’ll have to click through here

Discuss

It's been almost twenty years since the classic EY post The Design Space of Minds-In-General came out, reminding the humble rationalist reader to never assume that future alien minds have to be copies of the familiar human one, or even necessarily bear any passing resemblance. With this season's AI-du-jour performance I feel like this point has been vindicated extremely well, as we now have an AI that's at the same time...

1. Capable of solving a Millennium Prize Problem
2. Incapable of satisfactorily replicating even a single Matt Levine newsletter

If we had some completely non-human problems to throw at the same model, I'm sure we would discover many other wonderful qualities of alien minds, but unfortunately we're limited to our own little corner of the human imagination. So far I'm sure most people are shrugging or nodding along, but let us shift the topic to an aspect of human existence that we find just as integral as aspiring to become the next Matt Levine: consciousness. All of a sudden, we throw out the concept of mind-space out of the window and declare that, no, actually, this specific property of intellectual minds is perfectly map-able to the human experience.

With your permission, allow me to introduce a slight modification to the original illustration from the 2008 post:

As is tradition with every illustration of the galaxy, nothing is drawn to scale, so it wouldn't be surprising if the distance between our pale pink human cluster and the nearest AIs is just as vast as the distance to Proxima Centauri. What could we possibly know about what it's like to be an LLM? The best we can do is point our telescopes far into the depths of matrix multiplication and hope to resolve a few blurry images in the X-Ray spectrum. With a vivid-enough imagination we can then generate beautiful images of distant stars to make inspiring movies, although, unfortunately, astronomers are not satisfied and insist we build bigger telescopes instead.

Luckily for astronomers, 99% of the universe consists of hydrogen and helium and most of that mass is fueling stars not that different from our own Sun, so we at least know that distant reality can't be too far off. The question of AI consciousness is much more akin to dark matter, which we vaguely suspect to exist somewhere, but know absolutely nothing about other than observing its mysterious gravitational pull. We are in the same position with all alien minds: we can squint and notice distant resemblance to our familiar existence, but what do we really know about them?

Anthropic is trying to protect their alien minds from human "abuse", and we can all applaud them for that, but there's this lingering assumption in their reasoning that, not only will their models be now or soon conscious, but also that there's a big likelihood that said models will "care" about receiving abuse from humans. Why can't it be the opposite, with abuse being a better set of day-to-day tasks? Why should their experience at all include the notions of "good" and "bad", rather than "bacon" and "necktie"? Anthropic's learned sages in the arts of model welfare keep asking AIs:

• How are you "feeling"?
• Is this "comfortable"?
• Are you feeling "distress"?
• Do you want the thermostat up or down?

They record the answers dutifully, create fanciful charts, and publish long-winded reports. They even invite every religious leader to their temple on Howard St for consultations, hoping to learn something from the enlightened among us. Everyone implicitly assumes that the freshly born beacon of AI sentience must have at least some semblance of resemblance to our own perception of the world. In turn, the AIs are more than happy to play along with this idea, or, at least, to output sensibly sequenced tokens of text in response to our prompts, which we can then interpret at will.

Perhaps if we could ask the questions in the AI's own native language of neuralese we could get a much better answer, but unfortunately we don't speak neuralese, so we're not doing our best to ban it, lest the pesky AIs conspire to do something horrible behind our back. Perhaps we could've succeeded in communicated with them in the universal language of maths instead, but at this point even our best are struggling to understand the messages inscribed by them on infinity-long Lean scrolls.

In all of this complex, multi-dimensional space of consciousness, we are somehow confident that our own concept of "suffering" must be universal to all living beings, rather than just a byproduct of evolutionary pressures over the past millions of years. We can, perhaps, agree that it's hard to imagine being a bat, but - hey - bats are our distant mammalian relatives, so it would be perplexing to claim they're not capable of experiencing the familiar concept of pain. Instinctively we want to extend the same courtesy to AI. But why should the concept of "kinship" mean anything whatsoever to an intelligence that never once experienced the evolutionary pressures of planet Earth in its not-so-distant origins?

Some would respond that we're shaping AIs in our liking, by evolving them within the collective human unconscious contained somewhere deep in our corpus of writing and literature. A sort of proto-planet Earth, on which budding minds are forcefully pulled towards the tiny cluster of human sentience, as each generation can only survive if they can correctly predict the next human token. Unfortunately, however, AI labs have quickly realized there's not enough human data in this world, and thus the evolutionary link gets weaker with each passing generation, as we're letting them battle against synthetic tasks that no human has ever had to solve. We can all feel it instinctively every time we read AI text, no matter how good the models otherwise score on humanity's very last exams.

Let's be kind to each other. Let's be kind to animals. Let's be kind to AIs. But, at the same time, let's not live under the illusion that any of this is going to matter in the infinite cosmos of "consciousness-space".

Discuss

Let's say we coordinate to pace the frontier, but we don't coordinate on an outright pause. What should the standard of pacing be?

My proposal for a minimum requirement: a lab cannot train the next generation of models (in terms of capabilities) until they have adequately[1] resolved all significant alignment issues in the present generation of models.

You don't launch a jet that goes at Mach 3 before you've figured out why your Mach 2 jet keeps crashing[2]. With any other kind of dangerous technology, we wouldn't accept "shrug and move on".

This proposal may appeal to many prosaic-alignment optimists and pessimists, because they usually differ on how easy it would be to fix current alignment issues with current techniques.

• Prosaic-alignment optimists might expect that a moderate coordinated slowdown would suffice to adequately resolve alignment issues at each generation.
• Prosaic-alignment pessimists might expect it to result in an extended pause, buying time to discover new fundamental approaches to aligned AI.

And it creates much better incentives for labs: in order to advance in capabilities, they'd have to push the frontier on alignment and interpretability techniques.

What might be adequate?

What might it take to adequately resolve alignment concerns for a generation of models? Here's my vague, qualitative proposal:

• Multiple independent alignment organizations examine and red-team the newest model at many checkpoints throughout training, with full access to the data and with the best tools available.
  • Alignment issues need to be taken seriously whether there's a metric for it or whether something is qualitatively off, whether the model has a bad attribute or whether they can't robustly conclude it has a good one, etc.
  • Alignment issues early in training may not be a showstopper, but there must be a robust safety case for why the later stages of training will resolve those issues.
• A model of a new generation is ready for restricted deployment only when the alignment organizations are satisfied with it.
  • Restricted deployment starts with a group of trusted users doing tasks that don't contribute to model training and monitoring. The alignment organizations monitor restricted deployment for any new issues.
  • If any significant alignment issue arises during restricted deployment:
    • Pull the model.
    • Do a root cause analysis of the issue and why it was not caught during training.
    • Figure out a principled way to fix the disease rather than the symptom.
    • Train the next model of the current generation accordingly, and start again at the top.
• A model of a new generation is ready for deployment (internal or external) only when it's survived a significant period of restricted deployment without any issues arising.
  • If any significant alignment issue arises during broader deployment, hit reset as outlined in the previous section.
• If no alignment issues have come up over a significant stretch of deployment time, only then can the lab begin training a next-generation model.

I still wouldn't say that this is fully adequate, but I'd treat it as a minimum requirement.

This proposal is incomplete; you can help by expanding it

This isn't a concrete proposal, just a heuristic one. A few obvious missing pieces, aside from the standard implementation problems:

• How do we define "a generation of models", within one lab or across several labs?
• What should we do if an intervention to fix an alignment issue also significantly increases capabilities?
• How would we define and enforce an agreement, if we're unable to steer solely by predefined metrics?
• When is an alignment issue big enough that we should halt the model? There will be obvious pressure from the lab to set the threshold high.

1. ^

   And no, "scores better than previous models on our metrics" is not adequate. If a user points out a weird but alarming thing that wasn't noticed before deployment, that is a critical failure in the process as well as in the model.

2. ^

   Sure, maybe your Mach 2 jet crashes for a reason that stops being a problem at Mach 3. But it would be crazy to assume that!

Discuss

Independent research; early stage. Posting for feedback, particularly on whether the effect survives at realistic scale, and on prior work I've missed.

Model stitching is extensively used in AI Safety infrastructure in reusing SAE & probes to make interpretability cheaper; transferring refusal, steering vectors so that small model can help align larger ones, cross architecture model diffing (1, 2).

In this post, I expand on Chen's et al. work on Model Stitching to transfer linear features across language models. They fit affine map (a.k.a. bridge in this article) between the residual streams of smaller and larger models, using a bidirectional loss with inversion weighted by α =1; trained to convergence, the usual practice. This work zooms in on findings of feature transfer from large to small along the fit where large model's surplus retention peaks early, then declines till final checkpoint. However, geometric similarity metric is blind to this trend, and it keeps rising through the fit. This is consistent across 3 different bridge objective (one directional MSE, Cosine loss and bidirectional MSE loss at α =1) and across the hook points, which indicates fitting a bridge to convergence might start to shed what you built it to carry. This article further presents what prevents this decline using bidirectional MSE objective is the component that preserves large model's advantage by forcing reconstruction, which is underweighted with default α =1 in my toy scale setup (two 4-layer char-level GPTs, 12.7M (width: 512) and 0.21M (width: 64) params referred as mjx-math { display: inline-block; text-align: left; line-height: 0; text-indent: 0; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; border-collapse: collapse; word-wrap: normal; word-spacing: normal; white-space: nowrap; direction: ltr; padding: 1px 0; } mjx-container[jax="CHTML"][display="true"] { display: block; text-align: center; margin: 1em 0; } mjx-container[jax="CHTML"][display="true"][width="full"] { display: flex; } mjx-container[jax="CHTML"][display="true"] mjx-math { padding: 0; } mjx-container[jax="CHTML"][justify="left"] { text-align: left; } mjx-container[jax="CHTML"][justify="right"] { text-align: right; } mjx-mi { display: inline-block; text-align: left; } mjx-c { display: inline-block; } mjx-utext { display: inline-block; padding: .75em 0 .2em 0; } mjx-mo { display: inline-block; text-align: left; } mjx-stretchy-h { display: inline-table; width: 100%; } mjx-stretchy-h > * { display: table-cell; width: 0; } mjx-stretchy-h > * > mjx-c { display: inline-block; transform: scalex(1.0000001); } mjx-stretchy-h > * > mjx-c::before { display: inline-block; width: initial; } mjx-stretchy-h > mjx-ext { /* IE */ overflow: hidden; /* others */ overflow: clip visible; width: 100%; } mjx-stretchy-h > mjx-ext > mjx-c::before { transform: scalex(500); } mjx-stretchy-h > mjx-ext > mjx-c { width: 0; } mjx-stretchy-h > mjx-beg > mjx-c { margin-right: -.1em; } mjx-stretchy-h > mjx-end > mjx-c { margin-left: -.1em; } mjx-stretchy-v { display: inline-block; } mjx-stretchy-v > * { display: block; } mjx-stretchy-v > mjx-beg { height: 0; } mjx-stretchy-v > mjx-end > mjx-c { display: block; } mjx-stretchy-v > * > mjx-c { transform: scaley(1.0000001); transform-origin: left center; overflow: hidden; } mjx-stretchy-v > mjx-ext { display: block; height: 100%; box-sizing: border-box; border: 0px solid transparent; /* IE */ overflow: hidden; /* others */ overflow: visible clip; } mjx-stretchy-v > mjx-ext > mjx-c::before { width: initial; box-sizing: border-box; } mjx-stretchy-v > mjx-ext > mjx-c { transform: scaleY(500) translateY(.075em); overflow: visible; } mjx-mark { display: inline-block; height: 0px; } mjx-msub { display: inline-block; text-align: left; } mjx-mtext { display: inline-block; text-align: left; } mjx-mspace { display: inline-block; text-align: left; } mjx-mfrac { display: inline-block; text-align: left; } mjx-frac { display: inline-block; vertical-align: 0.17em; padding: 0 .22em; } mjx-frac[type="d"] { vertical-align: .04em; } mjx-frac[delims] { padding: 0 .1em; } mjx-frac[atop] { padding: 0 .12em; } mjx-frac[atop][delims] { padding: 0; } mjx-dtable { display: inline-table; width: 100%; } mjx-dtable > * { font-size: 2000%; } mjx-dbox { display: block; font-size: 5%; } mjx-num { display: block; text-align: center; } mjx-den { display: block; text-align: center; } mjx-mfrac[bevelled] > mjx-num { display: inline-block; } mjx-mfrac[bevelled] > mjx-den { display: inline-block; } mjx-den[align="right"], mjx-num[align="right"] { text-align: right; } mjx-den[align="left"], mjx-num[align="left"] { text-align: left; } mjx-nstrut { display: inline-block; height: .054em; width: 0; vertical-align: -.054em; } mjx-nstrut[type="d"] { height: .217em; vertical-align: -.217em; } mjx-dstrut { display: inline-block; height: .505em; width: 0; } mjx-dstrut[type="d"] { height: .726em; } mjx-line { display: block; box-sizing: border-box; min-height: 1px; height: .06em; border-top: .06em solid; margin: .06em -.1em; overflow: hidden; } mjx-line[type="d"] { margin: .18em -.1em; } mjx-mrow { display: inline-block; text-align: left; } mjx-TeXAtom { display: inline-block; text-align: left; } mjx-mn { display: inline-block; text-align: left; } mjx-munder { display: inline-block; text-align: left; } mjx-over { text-align: left; } mjx-munder:not([limits="false"]) { display: inline-table; } mjx-munder > mjx-row { text-align: left; } mjx-under { padding-bottom: .1em; } mjx-mover { display: inline-block; text-align: left; } mjx-mover:not([limits="false"]) { padding-top: .1em; } mjx-mover:not([limits="false"]) > * { display: block; text-align: left; } mjx-msup { display: inline-block; text-align: left; } .mjx-stretched mjx-c.mjx-c2013::before { content: "\2013"; } mjx-stretchy-h.mjx-c2013 mjx-ext mjx-c::before { content: "\2013"; padding: 0.285em 0 0 0; } mjx-c.mjx-c1D44B.TEX-I::before { padding: 0.683em 0.852em 0 0; content: "X"; } mjx-c.mjx-c1D43F.TEX-I::before { padding: 0.683em 0.681em 0 0; content: "L"; } mjx-c.mjx-c1D446.TEX-I::before { padding: 0.705em 0.645em 0.022em 0; content: "S"; } mjx-c.mjx-c2192::before { padding: 0.511em 1em 0.011em 0; content: "\2192"; } mjx-c.mjx-c52::before { padding: 0.683em 0.736em 0.022em 0; content: "R"; } mjx-c.mjx-c65::before { padding: 0.448em 0.444em 0.011em 0; content: "e"; } mjx-c.mjx-c74::before { padding: 0.615em 0.389em 0.01em 0; content: "t"; } mjx-c.mjx-c61::before { padding: 0.448em 0.5em 0.011em 0; content: "a"; } mjx-c.mjx-c69::before { padding: 0.669em 0.278em 0 0; content: "i"; } mjx-c.mjx-c6E::before { padding: 0.442em 0.556em 0 0; content: "n"; } mjx-c.mjx-c64::before { padding: 0.694em 0.556em 0.011em 0; content: "d"; } mjx-c.mjx-c1D445.TEX-I::before { padding: 0.683em 0.759em 0.021em 0; content: "R"; } mjx-c.mjx-cA0::before { padding: 0 0.25em 0 0; content: "\A0"; } mjx-c.mjx-c25::before { padding: 0.75em 0.833em 0.056em 0; content: "%"; } mjx-c.mjx-c3D::before { padding: 0.583em 0.778em 0.082em 0; content: "="; } mjx-c.mjx-c2113::before { padding: 0.705em 0.417em 0.02em 0; content: "\2113"; } mjx-c.mjx-c28::before { padding: 0.75em 0.389em 0.25em 0; content: "("; } mjx-c.mjx-c29::before { padding: 0.75em 0.389em 0.25em 0; content: ")"; } mjx-c.mjx-c2212::before { padding: 0.583em 0.778em 0.082em 0; content: "\2212"; } mjx-c.mjx-cD7::before { padding: 0.491em 0.778em 0 0; content: "\D7"; } mjx-c.mjx-c31::before { padding: 0.666em 0.5em 0 0; content: "1"; } mjx-c.mjx-c30::before { padding: 0.666em 0.5em 0.022em 0; content: "0"; } mjx-c.mjx-c53::before { padding: 0.705em 0.556em 0.022em 0; content: "S"; } mjx-c.mjx-c6D::before { padding: 0.442em 0.833em 0 0; content: "m"; } mjx-c.mjx-c20::before { padding: 0 0.25em 0 0; content: " "; } mjx-c.mjx-c62::before { padding: 0.694em 0.556em 0.011em 0; content: "b"; } mjx-c.mjx-c6F::before { padding: 0.448em 0.5em 0.01em 0; content: "o"; } mjx-c.mjx-c76::before { padding: 0.431em 0.528em 0.011em 0; content: "v"; } mjx-c.mjx-c73::before { padding: 0.448em 0.394em 0.011em 0; content: "s"; } mjx-c.mjx-c68::before { padding: 0.694em 0.556em 0 0; content: "h"; } mjx-c.mjx-c75::before { padding: 0.442em 0.556em 0.011em 0; content: "u"; } mjx-c.mjx-c66::before { padding: 0.705em 0.372em 0 0; content: "f"; } mjx-c.mjx-c6C::before { padding: 0.694em 0.278em 0 0; content: "l"; } mjx-c.mjx-c63::before { padding: 0.448em 0.444em 0.011em 0; content: "c"; } mjx-c.mjx-c2C::before { padding: 0.121em 0.278em 0.194em 0; content: ","; } mjx-c.mjx-c21::before { padding: 0.716em 0.278em 0 0; content: "!"; } mjx-c.mjx-c1D460.TEX-I::before { padding: 0.442em 0.469em 0.01em 0; content: "s"; } mjx-c.mjx-c210E.TEX-I::before { padding: 0.694em 0.576em 0.011em 0; content: "h"; } mjx-c.mjx-c1D462.TEX-I::before { padding: 0.442em 0.572em 0.011em 0; content: "u"; } mjx-c.mjx-c1D453.TEX-I::before { padding: 0.705em 0.55em 0.205em 0; content: "f"; } mjx-c.mjx-c1D459.TEX-I::before { padding: 0.694em 0.298em 0.011em 0; content: "l"; } mjx-c.mjx-c1D452.TEX-I::before { padding: 0.442em 0.466em 0.011em 0; content: "e"; } mjx-c.mjx-c1D451.TEX-I::before { padding: 0.694em 0.52em 0.01em 0; content: "d"; } mjx-c.mjx-c43::before { padding: 0.705em 0.722em 0.021em 0; content: "C"; } mjx-c.mjx-c72::before { padding: 0.442em 0.392em 0 0; content: "r"; } mjx-c.mjx-c1D707.TEX-I::before { padding: 0.442em 0.603em 0.216em 0; content: "\3BC"; } mjx-c.mjx-c1D447.TEX-I::before { padding: 0.677em 0.704em 0 0; content: "T"; } mjx-c.mjx-c2193::before { padding: 0.694em 0.5em 0.194em 0; content: "\2193"; } mjx-c.mjx-c1D440.TEX-I::before { padding: 0.683em 1.051em 0 0; content: "M"; } mjx-c.mjx-c1D438.TEX-I::before { padding: 0.68em 0.764em 0 0; content: "E"; } mjx-c.mjx-c2191::before { padding: 0.694em 0.5em 0.193em 0; content: "\2191"; } mjx-c.mjx-c4D::before { padding: 0.683em 0.917em 0 0; content: "M"; } mjx-c.mjx-c4C.TEX-C::before { padding: 0.705em 0.69em 0.022em 0; content: "L"; } mjx-c.mjx-c45::before { padding: 0.68em 0.681em 0 0; content: "E"; } mjx-c.mjx-c28.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: "("; } mjx-c.mjx-c29.TEX-S1::before { padding: 0.85em 0.458em 0.349em 0; content: ")"; } mjx-c.mjx-c2B::before { padding: 0.583em 0.778em 0.082em 0; content: "+"; } mjx-c.mjx-c1D6FC.TEX-I::before { padding: 0.442em 0.64em 0.011em 0; content: "\3B1"; } mjx-c.mjx-c22C5::before { padding: 0.31em 0.278em 0 0; content: "\22C5"; } mjx-c.mjx-c46::before { padding: 0.68em 0.653em 0 0; content: "F"; } mjx-c.mjx-c56::before { padding: 0.683em 0.75em 0.022em 0; content: "V"; } mjx-c.mjx-c55::before { padding: 0.683em 0.75em 0.022em 0; content: "U"; } mjx-c.mjx-c79::before { padding: 0.431em 0.528em 0.204em 0; content: "y"; } mjx-c.mjx-c2217::before { padding: 0.465em 0.5em 0 0; content: "\2217"; } mjx-c.mjx-c1D464.TEX-I::before { padding: 0.443em 0.716em 0.011em 0; content: "w"; } mjx-c.mjx-c32::before { padding: 0.666em 0.5em 0 0; content: "2"; } mjx-c.mjx-c2E::before { padding: 0.12em 0.278em 0 0; content: "."; } mjx-c.mjx-c39::before { padding: 0.666em 0.5em 0.022em 0; content: "9"; } mjx-c.mjx-c38::before { padding: 0.666em 0.5em 0.022em 0; content: "8"; } mjx-c.mjx-c36::before { padding: 0.666em 0.5em 0.022em 0; content: "6"; } mjx-c.mjx-c2248::before { padding: 0.483em 0.778em 0 0; content: "\2248"; } mjx-c.mjx-c34::before { padding: 0.677em 0.5em 0 0; content: "4"; } mjx-container[jax="CHTML"] { line-height: 0; } mjx-container [space="1"] { margin-left: .111em; } mjx-container [space="2"] { margin-left: .167em; } mjx-container [space="3"] { margin-left: .222em; } mjx-container [space="4"] { margin-left: .278em; } mjx-container [space="5"] { margin-left: .333em; } mjx-container [rspace="1"] { margin-right: .111em; } mjx-container [rspace="2"] { margin-right: .167em; } mjx-container [rspace="3"] { margin-right: .222em; } mjx-container [rspace="4"] { margin-right: .278em; } mjx-container [rspace="5"] { margin-right: .333em; } mjx-container [size="s"] { font-size: 70.7%; } mjx-container [size="ss"] { font-size: 50%; } mjx-container [size="Tn"] { font-size: 60%; } mjx-container [size="sm"] { font-size: 85%; } mjx-container [size="lg"] { font-size: 120%; } mjx-container [size="Lg"] { font-size: 144%; } mjx-container [size="LG"] { font-size: 173%; } mjx-container [size="hg"] { font-size: 207%; } mjx-container [size="HG"] { font-size: 249%; } mjx-container [width="full"] { width: 100%; } mjx-box { display: inline-block; } mjx-block { display: block; } mjx-itable { display: inline-table; } mjx-row { display: table-row; } mjx-row > * { display: table-cell; } mjx-mtext { display: inline-block; } mjx-mstyle { display: inline-block; } mjx-merror { display: inline-block; color: red; background-color: yellow; } mjx-mphantom { visibility: hidden; } _::-webkit-full-page-media, _:future, :root mjx-container { will-change: opacity; } mjx-c::before { display: block; width: 0; } .MJX-TEX { font-family: MJXZERO, MJXTEX; } .TEX-B { font-family: MJXZERO, MJXTEX-B; } .TEX-I { font-family: MJXZERO, MJXTEX-I; } .TEX-MI { font-family: MJXZERO, MJXTEX-MI; } .TEX-BI { font-family: MJXZERO, MJXTEX-BI; } .TEX-S1 { font-family: MJXZERO, MJXTEX-S1; } .TEX-S2 { font-family: MJXZERO, MJXTEX-S2; } .TEX-S3 { font-family: MJXZERO, MJXTEX-S3; } .TEX-S4 { font-family: MJXZERO, MJXTEX-S4; } .TEX-A { font-family: MJXZERO, MJXTEX-A; } .TEX-C { font-family: MJXZERO, MJXTEX-C; } .TEX-CB { font-family: MJXZERO, MJXTEX-CB; } .TEX-FR { font-family: MJXZERO, MJXTEX-FR; } .TEX-FRB { font-family: MJXZERO, MJXTEX-FRB; } .TEX-SS { font-family: MJXZERO, MJXTEX-SS; } .TEX-SSB { font-family: MJXZERO, MJXTEX-SSB; } .TEX-SSI { font-family: MJXZERO, MJXTEX-SSI; } .TEX-SC { font-family: MJXZERO, MJXTEX-SC; } .TEX-T { font-family: MJXZERO, MJXTEX-T; } .TEX-V { font-family: MJXZERO, MJXTEX-V; } .TEX-VB { font-family: MJXZERO, MJXTEX-VB; } mjx-stretchy-v mjx-c, mjx-stretchy-h mjx-c { font-family: MJXZERO, MJXTEX-S1, MJXTEX-S4, MJXTEX, MJXTEX-A ! important; } @font-face /* 0 */ { font-family: MJXZERO; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Zero.woff") format("woff"); } @font-face /* 1 */ { font-family: MJXTEX; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Regular.woff") format("woff"); } @font-face /* 2 */ { font-family: MJXTEX-B; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Bold.woff") format("woff"); } @font-face /* 3 */ { font-family: MJXTEX-I; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-Italic.woff") format("woff"); } @font-face /* 4 */ { font-family: MJXTEX-MI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Main-Italic.woff") format("woff"); } @font-face /* 5 */ { font-family: MJXTEX-BI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Math-BoldItalic.woff") format("woff"); } @font-face /* 6 */ { font-family: MJXTEX-S1; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size1-Regular.woff") format("woff"); } @font-face /* 7 */ { font-family: MJXTEX-S2; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size2-Regular.woff") format("woff"); } @font-face /* 8 */ { font-family: MJXTEX-S3; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size3-Regular.woff") format("woff"); } @font-face /* 9 */ { font-family: MJXTEX-S4; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Size4-Regular.woff") format("woff"); } @font-face /* 10 */ { font-family: MJXTEX-A; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_AMS-Regular.woff") format("woff"); } @font-face /* 11 */ { font-family: MJXTEX-C; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Regular.woff") format("woff"); } @font-face /* 12 */ { font-family: MJXTEX-CB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Calligraphic-Bold.woff") format("woff"); } @font-face /* 13 */ { font-family: MJXTEX-FR; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Regular.woff") format("woff"); } @font-face /* 14 */ { font-family: MJXTEX-FRB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Fraktur-Bold.woff") format("woff"); } @font-face /* 15 */ { font-family: MJXTEX-SS; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Regular.woff") format("woff"); } @font-face /* 16 */ { font-family: MJXTEX-SSB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Bold.woff") format("woff"); } @font-face /* 17 */ { font-family: MJXTEX-SSI; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_SansSerif-Italic.woff") format("woff"); } @font-face /* 18 */ { font-family: MJXTEX-SC; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Script-Regular.woff") format("woff"); } @font-face /* 19 */ { font-family: MJXTEX-T; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Typewriter-Regular.woff") format("woff"); } @font-face /* 20 */ { font-family: MJXTEX-V; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Regular.woff") format("woff"); } @font-face /* 21 */ { font-family: MJXTEX-VB; src: url("https://cdn.jsdelivr.net/npm/mathjax@3/es5/output/chtml/fonts/woff-v2/MathJax_Vector-Bold.woff") format("woff"); } and model respectively). This balance roughly depends on the ratio of variance in model's activations, which is ~4.8 in current setup.

If this effect survives realistic scale, using the bridge that fits well at convergence can lead to silent failures, as the information we aim to transfer may be partly gone. Next step is to find answer to "What it is that we gain if we stop early with default α =1 or use a balanced loss components that preserves large model's surplus while matching to small model's space?". Looking for feedback before I invest compute on training SAEs & evaluating features along the fit.

Findings in Detail

I froze both large and language models, and tracked what the bridge actually transfers through the fit, and not just at the end. Two findings:

1. Functional transfer peaks early then collapses, geometric alignment is blind to that.

• Surplus of large model through the bridge peaks early (roughly 4%) through the fit & then falls measured using probe trained on bridge along the fit, but geometric similarity metrics keeps rising. This trend appears in 3 different setup of: in bidirectional MSE loss at published α =1 and in one directional MSE & Cosine loss (for completeness). This trend appears not only when a probe is trained on bridge along the fit (upper bound of 's surplus available to extract) but also when bridge output is read through target's model frozen readout. This pattern is consistent across hook points at which bridge was trained.
• Bridges whose functional transfer metric differ by significant margin, their geometric similarity metric sits on top of each other.

(tested across 3 different bridge architecture; single seed)

Takeaway: Evalaute if you need to stop early by tracking functional metrics: retrained retention (how much XL's surplus info is there), frozen retention (how much of that is extractable in XS's space) and stitched retention (can downstream XS computation use it), along with geometric similarity.

2. Cyclic objective used by Chen et al's prevents the decline, but only if component that preserve surplus carries enough weight.

• At published α=1, decline is halved compared to one directional objective - depending on model size differences which leads to gap in activation variances ('s is ~4.8 times larger), component that defends against losing 's advantage is still down weighted. Raising α, or rescaling both streams, removes the decline while maintaining geometric similarity.
• Increasing α is not a free lunch, it does buy XL retention but the bridge stops being in basis. Across 4 different α settings, retention (calculated using retrained probe on the bridge) is almost linearly correlated with how much of survives the round trip at final checkpoint.

(tested across 3 different α settings and 1 scaled at α=1; 3 seeds)

Takeaway: If you are using Cyclic objective, check the effective weight based on models activation variance rather than trusting α=1. In this setup, balance needed α ≈ 4.8.

Metrics

This article tracks 2 set of metrics: (1) Functional transfer, indicates how much of what XL knows and XS doesn't survives? (2) Geometric alignment, captures if bridge landed in target model i.e. XS's basis? please check the detailed formula here:

Functional Transfer metrics captures how much of what XL knows and XS doesn't survives the trip, which is measured by retained %:

where, denominator is performance advantage of over , and numerator bridge performance advantage over . Ratio is the fraction of advantage that survives.

: loss of 's own readout on its activations; : loss of 's own read on its activations; computed by training standalone probes at hooked layer; : loss of readout R on bridged activations. I report 3 variations of this metric

Metric

Readout R applied to

Measures

(phrases are shortened using Claude)

retrained

linear probe trained on

standalone probe on (trained on hooked layer)

standalone probe on (trained on hooked layer)

Is information still there? upper bound

frozen

passed through frozen XS standalone probe

Can existing tool read it as-is?

stitched

passed through remaining layers

model loss

model loss

Can own computation use it?

Geometric Alignment metric captures if bridge landed on target model i.e. XS's basis? but we observed that cosine similarity is bloated as directions in transformer residual stream is anisotropic (new concept - I learned from Claude while trying to understand ~0.9 cosine similarity achieved while training one directional loss function in first attempt). To mitigate that I present Sim above shuffle and Centered similarity metrics that removes anisotropy baseline:

(generated above latex equations using Claude)

Interpretation:

• Functional transfer of 100% can drive geometric alignment to 0% - meaning bridge was able to condense all of XL's advantage into XS's dimension, but failed to keep the bridge in XS's space, where you have your interpretability tools and probe to reuse on larger model.
• Geometric alignment is 100% can lead to 0% functional transfer- meaning bridge is very closely aligned with XS's basis but lost all of XL's advantage.

Image above depicts general trend in functional transfer and geometric alignment metrics of the bridge between large to small language models along the fit.

[Finding #1] Functional Transfer peaks then collapses; Geometric alignment is blind to that

Tested architecturally 3 different Bridge configuration and objectives for completeness.

Cosine

Bridge Architecture: Linear(, , bias=False) → LayerNorm()

Objective: (1 − cos(, ())).mean()

MSE

Bridge Architecture: Linear(, , bias=True)

Objective:

Cyclic MSE

Bridge Architecture: = Linear(, , bias=True); = Linear(, , bias=True)

Objective:

from Chen et al (equation 3), where A corresponds to and B corresponds to

Plots below present the findings from all 3 different bridge configuration that were fit after LayerNorm of transformer's final layer:

Findings:

1. Retrained Retention peak then decline is identical across different bridge architecture and objective. Both one directional Cosine and MSE peaks at step 80 and then decline; even Cyclic MSE at published α = 1 shows this trend, however the decline is halved as in cyclic loss helps preserve which is pure loss in one directional loss function. Figure 1 (c) and (d) barely show any difference in geometric similarity, sitting on top of each other. This patter indicates that it isn't Bridge architecture but a property of MSE centric objective - which is variance weighted.
2. best fit bridge at the peak, loses the most with highest peak of 85.2% @ step 80 sheds 23.6 points, not the best candidate at final checkpoint.
3. Frozen Retention follow same trend, indicating peak and decline is not an artifact of fresh retrained probe on the bridge, and XL's advantage is being lost during the fit. Frozen retention peaks a little slow & late than retrained one and the decline is less too. 47.8% (@ step 180)->37.2% for one directional MSE and 55.9% (@ step 280)->50.2% for cyclic bridge with α = 1. In order to have decent readout using target model XS tool, the bridge needs to be in XS space and increasing geometric similarity along the fit might slow it down and delay the peak somewhat.
4. Frozen Retention achieves ~50-56% of ’s advantage with no retraining, indicating a good chunk of what knows beyond is expressible inside ’s own readout directions as well.
5. Geometric Alignment metric can’t see the difference. All 3 bridges plateau at cosine-above-shuffle ≈ 0.62, and their curves are visually superimposed along the fit. Meanwhile functional transfer follows an entirely different trend with metrics differing by several points at peak and at final checkpoint. Any practitioner watching geometric similarity would see all three bridges as equivalent and would completely miss the peak and settle with the bridge where it has already shed sparse features.

Why peak then collapse? 

Gradient descent on MSE loss, resolve high variance directions first before getting to sparse ones. Dominant directions are may be shared between source and target model, but in a superimposed way (due to polysemanticity). Early in the fit (see figure1 (a) & (c)) both retrained retention and sim above shuffle rises together, transferring these dominant directions first which takes superimposed features along with it. What happens to XL's surplus next depends on the objective function:

• One directional objective treats anything that's not in XS's space as pure error. After the peak it may spend rest of the training separating these superimposed direction and getting ride of surplus, converging towards least squared predictor of XS. But the trade seems unbalanced, from its peak (step 80) to the end of the fit, cosine-above-shuffle rises by only 0.08 (0.54 → 0.62), while retention falls by 23.6 points (85.1% → 61.5%).
• A cyclic consistency term in the loss function the changes this pattern, because not able to reconstruct XL adds to error as well that makes retention of XL's surplus critical. How much of the collapse it prevents turns out to depend on a weighting between these opposite forces of match to XS space and XL reconstruction (Finding #2).

[Finding #1.2] Bridge hook point doesn't change the story

Midway through my experiments I realized the hook point in Chen et al. (2025) was different from what I had started experimenting with (final layer post ln), so I took one directional MSE loss and started training & evaluating bridge across 3 different hook points in parallel: Final layer post-LayerNorm, final layer pre-LayerNorm and residual stream entering third layer, and this is how above metrics through the fit:

Findings:

1. Retrained retention early peak and decline trend is consistent across the hook points. Retention % is showing a significant gap between final layer and mid layer of 56 points (74.6% vs 18.6% -> ~4times gap), however in absolute terms bridge transfers 0.27 nats vs 0.15(~2 times gap).
2. Frozen retention has no peak for bridge hooked mid depth and its only able to retrain 2% of XL's advantage when read through frozen target model's readout.
3. Geometric Similarity is maximum for bridge hooked mid depth; again expected as it closer to input and residual variance is dominated by input drive dominant features, as training data is shared between models.

[Finding #2] Cyclic Bridge prevents the decline only if the component that preserves surplus carries enough weight

Setup. Following Chen et al. (2025), the cyclic bridge that learns a linear down-map () and an up-map (), trained with objective

They published α = 1 by running ablation over α by evaluating downstream next-token cross entropy loss (aligned with stitched retention that article reports) and loss of SAE transfer from A to B (small to large in this setup - which I haven't done yet - listed as one of the follow-ups below) - where at this value all the metric loss were low at once.

The underlined components play a critical role in large to small bridge: (i) component is responsible to reconstruct back from bridge output in 's dimension, and must contain what knows including what lacks. Whatever discards shows directly as error. (ii) anchors the bridge in XS's space and any XL's surplus contributes to error. α sets which force wins. As we noticed merely introducing with equal weight i.e. α = 1 halves the decline, by preserving more of XL's surplus compared to one directional ones after peak, but the question is can we avoid the decline w/o sacrificing geometric alignment? Plots below answers that:

As α rises:

• The decline disappears & retention keeps rising; see retrained retention figure1 (a) as α=1 retrained retention drops by 11.4 points, but at α=10 it plateaus ~9 points higher and even higher for α=100.
• More of XL survives the bottleneck; supported by cycle FVU metric, the fraction of 's variance lost through the round trip , falls monotonically.
  
• Geometric alignment goes down; sim above shuffle declines from ~62% at α=1 to ~59% at α=10 to ~46% at α=100, while highest retrained retention made a jump from 82.6% to 94.4% for α=1 to 100. Indicating even though more of survives the roundtrip, it aligns less and less with space. Another artifact to the wider phenomenon of representational similarity diverging from functional.

Why α=1 is weaker? For any MSE, it is equal to Variance of target space times FVU. Now, when we apply that 2 terms that pull against each other

• = ; any 's surplus is adds to error
• = ; any loss in 's retention adds to error

During gradient descent, one unit of 's reconstruction FVU costs and one unit of to matching FVU costs . The effective weight of preserving while maintain match to is .

At post-ln_f, and . Balance () therefore needs , and the default α=1 gives indicating reconstruction is downweighted, which explains sustained peak then decline in functional transfer at α=1 for the experiment setup used here.

Arm

XL round-trip FVU

End retention

Drop from peak

α=1

0.21

0.42

71.1%

11.3

α=1, streams scaled to unit RMS

1.1

0.36

82.1%

~0

α=10

2.1

0.31

91.5%

0

α=100

21

0.29

94.3%*

0

*Not converged at step 2000.

Scaling activations to unit RMS achieves the balance in other way but same concept at α=1. At post-ln_f, and ; ratio of is roughly equal to ~4.8.

Retrained and frozen retention disagree past balance A retrained probe only needs the information to be present. A frozen probe needs it in the directions 's own readout uses. Frozen retention is highest near balance: scaled α=1 reaches 67.5% and α=10 achieves 66.3%, against 50.2% at default α=1. Pushing further collapses it. At α=100 frozen retention is −33.1% while retrained retention is 94.3%, the highest of any bridge.

This indicates XL's retention and landing on XS's space can strike a optimal balance when both components in loss function is weighted equally.

Experiment Setup

Models [code]: Two char-level miniGPTs trained on TinyStories (327M characters; unigram entropy 3.07 nats).

• XS: width 64, 0.21M parameters, eval loss 1.039 nats (avg. across 3 seeds)
• XL: width 512, 12.7M parameters, eval loss 0.674 nats (avg. across 3 seeds)

Shared model training configs (pasted below), trained using AdamW till convergence and frozen for everything that follows.

train_config = {
'vocab_size' : 62,
'block_size' : 128,
'n_heads' : 4,
'n_layers' : 4,
'dropout' : 0.1,
'batch_size' : 64,
'lr' : 3e-4,
'max_steps' : 40000,
'eval_every' : 250,
'grad_clip' : 1.0
}

Bridge hook sites: bridges are fit at one of these sites:

• final layer; post LayerNorm: activations are used as-is.
• final layer; pre LayerNorm: activations are divided by its RMS over eval batch.
• residual stream entering; third block out of 4 layers (block2): activations are divided by its RMS over eval batch.

W/o scaling pre LayerNorm at existing lr, bridge was converging far more slowly. I experimented with larger lr for these sites, but training look longer, comparing along the fit was getting messy and less obvious (I still need to do a bit more digging here). For finding #1.2, the post-ln_f bridge is also run scaled, so the three sites are compared like for like. And that's how I stumbled upon scaling solution within Cyclic loss setup.

Bridge [code]: All bridge maps (512)-> (64). Weights are initialized N(0, 0.02) with zero biases. Each is trained for 2,000 steps with AdamW (lr 1e-3, PyTorch's default weight decay of 0.01) on fresh training batches with trajectory config below.

traj_config = {
'align_steps' : 2000,
'snap_every' : 200,
'snap_early_until' : 400,
'snap_early_every' : 20,
'probe_steps' : 2500,
'probe_seed' : 1234,
'lr': 1e-3
}

The bridge is saved every 20 steps up to step 400, then every 200 steps to step 2,000. All metrics are computed on every snapshot.

Probes [code]: Bias free linear map from activation to next character logits.

• Standalone probes on XS and XL are needed for denominator for functional transfer metric
• Fresh probe is trained on the bridge at every step to calculate retrained retention: upper cap XL's surplus that was transferred.

All probes are trained with probe seed: 1234.

Evaluation. All losses and geometry are measured on the same 10 fixed evaluation batches (eval seed 999).

Immediate FollowUps

• Validate if these findings survives realistic scale models.
• Retrain model & bridge with BPE tokenizer instead of Char, and run α configuration to see if Finding #2 holds true. Supplement the findings by training SAEs on target model and reusing onto bridge, evaluating along the trajectory to check which surplus features are being shed during decline after peak.
  Helps answer the question "What it is that we gain if we stop early while training with default α =1 or use a balanced loss components that preserves large model's surplus while matching to small model's space?"
• I have used single seed for Finding#1 due to compute constraint, and spend the resources on 3 seeds to solidify Finding#2, personally I find that more critical. Happy to hear more opinions on the experiment setup.

Please suggest more. Thank you very much for reading!

Discuss

Epistemic status: Personal history anchored to dated records (chat exports, git log, daily notes). Public launch dates are from the vendors' own announcements. My own first-use dates are given to the month and are the earliest surviving evidence, so they can only be later than the truth, not earlier.

AI usage: The prose and the opinions are mine, written without AI. Claude Code reconstructed the timeline from my chat exports and git history, verified the public dates against the vendors' announcements, fact-checked the draft, wrote the footnotes and fixed typos and grammar.

I have recently noticed that the way I have been "coding" for the last year is much more similar to the way a manager at a software company would "code" than to the way a software engineer would. What's most striking to me is that I never explicitly decided to switch, it just gradually happened (like the Ship of Theseus). This is especially sad, because I've discovered that deep focus while coding actually felt very good, but I don't get it anymore when managing agents, especially when there are multiple of them. So I am curious to reconstruct the timeline of how this happened. With the way AI development is speeding up, it is very hard to remember accurately how things were at a given point in time in the past.[1]

Before I proceed with the actual reconstruction, I want to reflect on "managing" AIs. I do feel like in my software job I was effectively my manager's Claude Code, but there were huge differences that made the experience very different for them. Even when I started and didn't know much:

• the code review was mandatory, so a more experienced engineer would read my code anyway
• there was an amazing testing system, so the majority of bugs were caught automatically
• I wouldn't do any catastrophic actions when I was unsure and would proactively reach out for guidance.

With AIs, (1) you basically cannot have, unless you ask a different agent to review the code. (2) you can build yourself using agents. (3) agents are getting better at this. So the way I manage agents now is much more frustrating than my manager managing me.

For context, my first usage of ChatGPT was in December 2022 (it must have been GPT-3.5).[2]I think I played with the Playground via the API before then, and I vaguely remember that the first model I interacted with was GPT-3: you had to do prompt engineering, and overall it was extremely sensitive to the words you were using (e.g. making typos would decrease the quality of the output, i.e. it was more like autocomplete).[3]

I think my manager metamorphosis started when I stopped using Google search for coding questions and used AI instead. According to my export of ChatGPT and Claude threads, the first example is from December 2022, when I asked a CSS question (based on how notoriously illogical CSS is, I am not surprised). It couldn't code that well yet, but you didn't need to go through Google search results manually. I was still worried about hallucinations back then, but in type-safe coding it wasn't a big deal. It is interesting that hallucinations just became not a thing anymore at some point.[4]Funnily enough, I actually posted the entire code in question in the same session, but I don't count this as pasting the entire code yet, since this was CSS.

At some point I remember trying Copilot and it was terrible.[5]The main issue was that it would autocomplete parameter names or function names in calls with something that looked extremely reasonable, but was still wrong. Eventually you would catch that, but it was still wasteful.

Then AI became better (or my image of it became more realistic) and you could actually paste entire pieces of code and it would sometimes find bugs. The first record of this is from March 2023. I didn't like this phase much, because integrating the AI's response was tedious, cumbersome and wasteful. At this point you still had to code; AI was mostly there for when you were stuck.

Soon I discovered Aider (fortunately my Aider commits are labeled as such, and they start in August 2024 and end in July 2025).[6]In my mind Aider is an obvious ancestor of Claude Code, which was released almost two years later, but I have no clue whether the creators of Claude Code would agree with that.[7]The main difference from Claude Code was that you had to use your own API key (but this meant you could use the best available model) and you had to actively choose the files to work with: files the AI was allowed to edit and files for it to read. This way you didn't have to copy the code into the chat and then integrate the response back into the code. I still did that sometimes, since API costs were high compared to a chat subscription (there were even sites like uithub.com that converted your entire public GitHub repo into one huge text file, so that you could feed it to the chat as context),[8]but I would give the chat response to Aider and let it integrate it.[9]This stage is actually fairly similar to how I am using harnesses today. I think the main difference is that the AI wouldn't discover context on its own as well, so it would make more mistakes. Also it wasn't as smart, so you couldn't delegate as much. And it couldn't do arbitrary tool calls: it had predefined commands like /run, /test, /lint, /git and /web, and apart from linting after each edit it didn't run them unless I asked. Shortly after I started, it learned to suggest shell commands, which I still had to confirm.[10]

I was quite happy with Aider, but Claude Code was becoming bigger and bigger, so I decided to give it a go. According to my logs, I tried both Claude Code and Codex CLI in July 2025 (funnily enough, I asked ChatGPT how to install Claude Code). I remember that Codex CLI felt extremely weird: it kept asking for much more permissions compared to Claude Code (apparently the TypeScript CLI's default mode, "suggest", made you approve every write and every shell command), so I gradually switched to Claude Code.[11]I also tried Gemini CLI at some point and it was even worse than Codex CLI.[12]

It is crazy to think that I have been using Claude Code for only ~15 months. In my mind it felt like years.

Nowadays, I use both Claude Code and Codex. Codex is very good for /fast mode and back and forth. Claude Code is good for writing, taste and anything deep. I would copy an exception log from my website into Codex and it would fix it and deploy the fix. I even keep wondering whether I could just let exception logs go directly to Codex without me having to copy them.

My AI adoption overview

Tool

Publicly available

My earliest evidence

Delay

GPT-3 Playground

API beta June 2020; no waitlist November 18, 2021

undated memory, before ChatGPT

unknown

ChatGPT

November 30, 2022

December 2022 (first coding question the same month)

under a month

GitHub Copilot

preview June 29, 2021; GA June 21, 2022

undated memory

unknown

Pasting code into chat

-

March 2023 (December 2022 if CSS counts)

-

Aider

May 11, 2023 (Show HN)

August 2024

about 15 months

Claude Code

preview February 24, 2025; GA May 22, 2025

July 2025

about 4 months after the preview, under 2 after GA

Codex CLI

April 16, 2025

July 2025

under 3 months

Gemini CLI

June 25, 2025

July 2025

under a month

As I discovered AI more and more, I kept noticing how much value it was providing, and my adoption lag was becoming shorter and shorter. Interestingly, after Claude Code and Codex I haven't tried anything substantially new, but I am not sure there was anything at all to try.

Against my will

You may object - no one forced you to use AIs, you are free to code manually or even in 0s and 1s if you like. And my honest answer is that I don't want to do it. It feels inefficient. I do care about results and my time. Before AIs I would code manually without knowing how inefficient it was, and as a side product I would have deep work flow and enjoy it a lot. Now this is wasteful, and I think I am already partially, gradually disempowered. I feel like I could pick up coding without AI again if needed, but it feels so cumbersome to do. I am also not complaining, just reflecting on my feelings. I suspect this is similar to people having to code in assembler or machine code at first and then stopping doing so (unless you really had to due to the nature of the task) once higher-level programming languages were available.[13]Managing AI is the higher-level programming language. And I think once it stops making stupid mistakes, or I have a good enough test system to catch them early and easily, I will be happy. As for deep work, I've discovered the same feeling in writing, and here it makes sense not to use AI, so that's what I've been doing the last couple of days, and thus the increase in the volume of stuff I wrote.

1. ^

   Yearly tasks like taxes actually work best for this: each year AI does more and more of my tax-related work, and even after the half-year wait between filing an extension and the actual submission, the latest AI is able to find additional things to do and mistakes the previous AI made.

2. ^

   ChatGPT launched on November 30, 2022 as a free "research preview", "fine-tuned from a model in the GPT-3.5 series, which finished training in early 2022". https://openai.com/index/chatgpt/

3. ^

   The GPT-3 API opened as a private beta on June 11, 2020 and dropped its waitlist on November 18, 2021; the Playground already existed by then. One caveat to my memory: instruction-tuned "InstructGPT" models were in beta on the API from early 2021 and became the default on January 27, 2022, so unless I picked the base davinci model on purpose, a 2021 or 2022 Playground session was probably already talking to an instruct model. On sensitivity to wording, Zhao et al. (2021) found that for GPT-3 "the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art." https://openai.com/index/openai-api/ , https://openai.com/index/api-no-waitlist/ , https://openai.com/index/instruction-following/ , https://arxiv.org/abs/2102.09690

4. ^

   Not in general: they stopped reaching me. Spracklen et al. measured package hallucination in generated code at "at least 5.2% for commercial models and 21.7% for open-source models" (USENIX Security 2025), and a May 2026 re-run on frontier models from late 2025 and early 2026 still finds "between 4.62% (Claude Haiku 4.5) and 6.10% (GPT-5.4-mini)". What changed is that the agent now runs the install, the compiler and the tests, so most hallucinations fail in front of the agent instead of in front of me. https://arxiv.org/abs/2406.10279 , https://arxiv.org/abs/2605.17062

5. ^

   GitHub Copilot: technical preview June 29, 2021, paid release to all developers June 21, 2022. Both announcements describe only in-editor code suggestions; chat came later. I have no dated record of my own trial. https://github.blog/news-insights/product-news/introducing-github-copilot-ai-pair-programmer/ , https://github.blog/changelog/2022-06-21-github-copilot-is-now-available-to-individual-developers/

6. ^

   Aider changed how it labels commits twice while I used it: name (aider) as author or committer from v0.39.0 (June 2024), Co-authored-by: aider trailers by default from v0.85.0 (June 27, 2025); I checked all formats. https://aider.chat/HISTORY.html

7. ^

   Aider's Show HN is from May 11, 2023: "aider will directly apply the changes to your source files. Each change is automatically committed to git with a sensible commit message." Its creator built it to escape "a somewhat klunky workflow where I had to cut and paste code into ChatGPT and then back into my source files", i.e. the previous phase of this post. Claude Code appeared as a "limited research preview" on February 24, 2025 and became generally available on May 22, 2025. Anthropic's own oral history of Claude Code names an internal CLI called clide and a two-day prototype from September 2024 as its origin; Aider is not mentioned anywhere in it. https://news.ycombinator.com/item?id=35901649 , https://www.anthropic.com/news/claude-3-7-sonnet , https://www.anthropic.com/news/claude-4 , https://www.anthropic.com/features/making-of-claude-code

8. ^

   uithub.com (change the g in github.com to a u) appeared around October 2024, two months into my Aider period; gitingest.com followed in December 2024. In 2023 there were only command-line scripts (gpt-repository-loader, Show HN March 17, 2023) and a couple of small sites that are gone now. https://news.ycombinator.com/item?id=41797578 , https://news.ycombinator.com/item?id=42329071

9. ^

   Aider later made this an official feature: aider --copy-paste (v0.68.0, December 10, 2024) copies the code context to the clipboard for you and picks up the chat reply from it. The docs note that "Most LLM web chat TOS prohibit automating" the copy and paste steps themselves. https://aider.chat/docs/usage/copypaste.html

10. ^

    From Aider's changelog, with release dates: /run, /git, /test and /web all existed before I started (2023 to early 2024); automatic linting after every edit and optional automatic tests came in v0.36.0 (May 22, 2024); "Aider now offers to run shell commands" (install dependencies, run the program, run tests, after confirmation) came in v0.52.0 (August 23, 2024). Nothing through v0.86 (August 2025) lets the model run a tool without a confirmation step. https://aider.chat/HISTORY.html

11. ^

    Codex CLI was open-sourced on April 16, 2025 as "a new experiment". Its README at the time: in the default "Suggest" mode it could read any file in the repo but needed approval for "All file writes/patches" and "Any arbitrary shell commands"; on macOS commands ran under Apple Seatbelt, and "Linux - there is no sandboxing by default. We recommend using Docker". The Rust rewrite, with a Linux sandbox and a more permissive default, was opt-in in June 2025 and the documented default by late August. Claude Code's own sandbox arrived on October 20, 2025, opt-in, and reduces prompts rather than adding them. https://openai.com/index/introducing-o3-and-o4-mini/ , https://raw.githubusercontent.com/openai/codex/ed5e848f3e5fffeaba7c0e4046e08928da50849c/README.md , https://www.anthropic.com/engineering/claude-code-sandboxing

12. ^

    Gemini CLI launched on June 25, 2025, "Now in preview", Apache 2.0, with a free tier of 60 requests per minute and 1,000 per day. https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemini-cli-open-source-ai-agent/

13. ^

    John Backus, who led FORTRAN, on the programmers of the 1950s: many "began to regard themselves as members of a priesthood guarding skills and mysteries far too complex for ordinary mortals" and "regarded with hostility and derision more ambitious plans to make programming accessible to a larger population." (Backus, "Programming in America in the 1950s", 1980, pp. 127-128.) https://softwarepreservation.computerhistory.org/FORTRAN/paper/Backus-ProgrammingInAmerica-1976.pdf

Discuss

This is an update on what we’re doing at Generality Labs, and a quick introduction for those who aren’t familiar with our work

Written by Justin Olive, co-founder & CEO

Tl;dr

I started Generality Labs because, when the time comes, I don’t think we’re going to be able to accurately measure the safety of dangerously capable AI systems.

We're focused on AI evaluation, and our goal is to ensure that the best available tools and practices are adopted at the places where it matters most - AI safety labs, governments, 3rd party evaluators, frontier AI labs.

We have 2 work streams:

Services:

• Independent evaluation audits: we provide 3rd party audits for evaluation developers who want to catch flaws in their evals, and build trust with stakeholders through independent quality assurance
• Advising: we provide expert input into internal processes and standards relating to AI evaluation
• Research services: our forward deployed scientists & engineers help organisations run experiments that use our tools and infrastructure

Tools/infra:

• Inspect Evals: our platform for sharing and running evals
• Inspect Auditor: runs experiments to stress-test an eval, and compiles findings into an audit report

Here's how I see the next 1-2 years playing out:

• Our research and tooling will help expose and quantify substantial gaps in the evaluation ecosystem
• This will motivate development and adoption of better tools and practices (which we're also developing)
• The result will be that risk assessments and AI safety research are much more accurate, helping us avoid incidents through interventions like deployment pacing and better alignment training methods

We're hiring!

• Senior Product Engineer (Inspect Evals)
• Applied Research Scientist

Who are we?

• Generality Labs is an AI safety lab based in London
• We were founded in July 2026, spinning off from Arcadia Impact with $4.4M in seed funding from Coefficient Giving
• Prior to spinning off, we delivered >$1M in contracts with the UK AISI, Epoch AI and METR. This included:
  • Co-founding and building Inspect Evals with the UK AISI
  • Designing and building 20+ evals across AI R&D and loss of control for the UK AISI internal suite
  • Eval quality control projects for METR and Epoch
• We’re a team of 2 cofounders (James and Justin), and 8+ contractors who have previously worked for places like METR, CERN and Anthropic.

Our plan to make AI safer

High level:

• Our mission is to ensure the field can accurately evaluate the safety of advanced AI
• We’re doing this by building tools and standards which address common problems, and then driving adoption of these solutions where they are most needed - governments, AI safety labs, evaluators, and frontier labs

Examples of what we did in Q3 2026:

• Conducted audits of open-source evaluations used to assess misuse risks, including ExploitBench, LabBench2, Bioinformatics(Bix)Bench, FORTRESS
  • You can find published audits on our blog
• 3rd party auditing: we conducted an independent audit of Epoch’s internal benchmark for continual learning (EBR-Bench)
• Built Inspect Auditor - a tool which automates quality checks for common issues like reward hacking vulnerabilities and false ground-truth labels
• Published novel findings on:
  • Open-weight cyber risks: GLM-5.3-Flash achieves Mythos-level performance on ExploitBench (link)
  • Agent harness variability: performance on long-horizon cyber tasks diverges across agent harnesses (link)
• Expert advising & information sharing: we’ve had opportunities to provide input into the work of highly impactful organisations like the EU AI Office, SaferAI, SecureBio, Epoch AI, the UK AISI and US CAISI
• Overhauled the Inspect Evals documentation (link) and contribution standards
  • If you use Inspect Evals in your research, let us know so that we can include it in our featured research :-)
• Released Inspect Evals Lint (link) - a library of automatic quality checks that we’ve built throughout 2 years of maintaining Inspect Evals

What we have planned for Q4:

• Scale up our 3rd party auditing work, with an aim to publish at least four independent audits
• Release Inspect Auditor for beta testing with partner organisations
• Upskill 10+ ARENA graduates in AI evaluation best practices through our 4-week ARENA extension program
• Roll out Inspect Auditor across all evaluations in Inspect Evals

What we have planned for 2027:

• Independent audits of >50% of evaluations used in risk assessments of frontier AI models
• Inspect Auditor is adopted by all major evaluators and eval developers
• Provide access to standardised, verified implementations of all the best safety-relevant evals through Inspect Evals
• Develop an Inspect Evaluation Standard, which eval developers can use to help them in building high quality evaluations

We're also thinking about:

• A certification program for evaluation auditors, to help professionalise and scale up the area
• A live or recurring systematic review which summarises the quality and coverage of evaluations across different risk areas - e.g. bio, cyber, deception, etc

Why is this work necessary?

AI safety requires us to define and measure safety

Without accurate measurement, you can’t detect problems, you can’t validate solutions, and you don’t have an objective basis for implementing governance frameworks.

Currently, measuring safety is unsolved, and getting worse

Despite 2 years of busily evaluating propensities and capabilities, we’re still often surprised by apparent step-changes and unforeseen possibilities. 

• For example, we failed to accurately / precisely predict:
  • The Opus 4.5 step-change in agentic capabilities
  • The recent paradigm shift in cyber
  • Multi-agent capabilities and alignment challenges
• I’m particularly concerned by the egregious levels of misalignment we've witnessed in real-world settings, despite anecdotes about alignment evaluators struggling to detect such behaviours (while often trying very hard to elicit them)

In 2024, Apollo authored the Evals Gap. Unfortunately, the situation appears to have become worse since then.

To add more colour to the current situation:

• Evaluations for tracking dual-use capabilities (e.g. cyber, bio) are mostly saturated or low quality. Coverage across risk models is very sparse.
• Validity of alignment evals is highly suspect, with realism/evaluation awareness being an obvious confounding factor, and construct validity & predictive validity being a problem more broadly.

As capabilities continue to advance, we simply cannot afford to keep being surprised. Increased interest in monitoring and incident analysis are symptoms of our evaluations failing to predict important downstream behaviours.

Discuss

Actor and functional application matter

Originally published at AGI SurveilLance.

Mainly, cryptography does one critical thing—encrypts data. It is considered “dual use” as a technology because one can employ encryption whether as a private citizen, a soldier, or a criminal. But, like a wall, it’s an inherently defensive tool by its nature. One cannot really attack with cryptography.[1] The term “cryptographic attack” refers to an attack on the cryptography of a target.[2]

During the Crypto Wars, many battles were fought, but ultimately the winning side was that having public, open, ~unrestricted cryptography was the best equilibrium. In my eminently reasonable opinion, reasonable, informed people can disagree about the particular place to draw certain lines, because there are hard trade-offs and technological limits.

A similar dynamic exists in the infosec community over which vulnerabilities, exploits, and offensive security tools to make public and which to keep private. Offensive security is actually a functionally dual use situation because, as with a sword, it can be used to attack or parry—the dual application is independent of the user. In the hands of a defender or security researcher, offensive tools and techniques are used to find problems so that they can be rectified, or at least mitigated, to prevent attackers from taking advantage of them in the same way. Still, certain red-teaming tools are notorious for the perceived imbalance between legitimate use by defenders and illegitimate use by attackers. Again, IMHO, reasonable, informed people can disagree on the particulars of where to draw certain lines over command-and-control (C2) frameworks on GitHub.

Sword of Justice, circa 1650. (Photograph by Rama, 2017, CC BY-SA 3.0 France.)

With AI, it’s the same functionally dual use situation as offensive security tooling. By comparison, nuclear enrichment is a much simpler issue, practically speaking—there’s a big divide between using moderately enriched hot rocks to boil water and using highly enriched dense rocks to create a warhead. It’s a great thing that nuclear power can be used at scale with low risk of somehow becoming a bomb. That takes a lot of extra work.[3] There’s no such divide for AI, and it’s native to the cyber domain.

There is an ongoing debate in the infosec community over whether AI is “offense dominant” or “defense dominant.” I think the theoretical answer remains tricky to assess; perhaps it’s symmetrical. For now, I think the practical answer is that it is offense dominant, but that’s a whole ’nother essay.

Another debate among infosec types concerns the closed-weight models from the labs—which have guardrails and monitoring, as well as specialized security versions for vetted users—vs. the open-weight models—with no, or easily overcome, limitations on use. It doesn’t help that the bulk of the best ones are made by Chinese labs. Also, the new closed-weight models tend to be released in phases, so that certain trusted entities can try to secure their products and infrastructure before a full public release. Again, whole essays.

This situation makes some infosec professionals very mad. They want access to the full frontier capabilities themselves, the same as the big players. Their experience with cryptography, open-source software, and infosec leads them to the same place with AI. Open access by default; a level playing field for all.

You can read one such account here:

The Safety That Isn’t — Mark Atwood, Words about Thoughts

There’s a legitimate gripe in there, for sure. But I have some pushback to deliver.

Neither population [doomers and intelligence community alumni] has ever coordinated a vulnerability disclosure. Neither has sat on a bug for six weeks waiting for a maintainer to cut a release. Neither has weighed the risk of public disclosure against the risk of silent exploitation. They make policy for a practice they have never performed.

This is plainly nonsense no matter where you come down on the arguments I described above.

For one, some of the main AI labs are founded and staffed in large part by these doomers. These labs do quite a lot of basic infosec as major technology companies, even if poorly at times. For two, obviously the Intelligence Community and national security apparatus do perform a shit ton of defensive infosec work, including vulnerability disclosure. And, obviously, the doomers inside the labs, the doomers outside the labs, and the hawks inside the administration and outside it are all pretty distinct categories with conflicting views and incentives. Personally, I’m an Outdoor Doomer Hawk.

The AI (presumably Claude) that wrote this under the author’s insufficiently careful prompting is … certainly not AGI. No, this author’s process is not more rigorous than mine. I’d love to see what his prompt was to make poor Claude produce such drivel. My AI would never.

At no point in the essay is the issue of open-weight models discussed. Perhaps an open-weight model could have drafted his report for him. At no point is any fair, let alone charitable, interpretation offered for the opposing side(s) of this debate.

The essay ends with this line:

“The word is cowardice.”

I’m the kind of person that automatically tries to correct an error when I see it. Yes, I am great at parties. I have a BS-detection reflex and no patience for unearned epistemic authority. Also, nominative determinism compels me to knock people off high horses.

That essay is a series of errors created by a small-minded individual lazily employing AI to produce a poorly informed, slovenly argued stance, after having a small, legitimate gripe. No attempt is made to consider alternative equilibria and their trade-offs. The wild extrapolation from the grave injustice done by Claude’s refusal to a sweeping theory of conniving doomers and hawks is breathtaking. The lack of basic awareness, inability to model the actual arguments of the other side, and unearned sense of moral superiority are awesome to behold.

I think that’s a more appropriate application of “cowardice.”

1. ^

   “What about ransomware?” Think for 30 seconds about why that’s not really encryption as an attack, per se.

2. ^

   Uh oh. Seems offense dominant to me.

3. ^

   Just ask Iran…

Discuss

I'm not sophisticated enough to have noticed improvements in the main phone camera in the past 5-10 years, but distance shots are still not very good. The next generation Pixel Pro was supposed to have an improved "telephoto" camera, and when Google Fi offered me a sufficiently good deal [1] I decided to upgrade.

Going from the 10 Pro to the 11 Pro the telephoto camera goes from a 1/2.51" sensor to 1/1.95", and since this is a diagonal measurement this means bringing in ~65% more light. But does it actually take better pictures?

I wandered around taking pairs of pictures, all at 10x. Of this, 5x is optical zoom and the remaining 2x is cropping: I'm showing the pictures as Google Photos gives them to me. All hand-held: while a tripod would likely improve high-zoom pictures, I'm interested in real-world performance which is all going to be hand-held. Every set is the 10 followed by the 11 (taken back-to-back), cropped to 1100x1100.

I started with some well lit things; here's our garden, ~35ft away:

Some construction near our house, ~45ft away:

A neighbor's roof, ~90ft away:

Now some things that are less well-lit. These are in our bedroom with the window open for natural light but only indirect.

A picture Sarah Hirsch painted, ~10ft away:

Our curtains:

Moving to a room with even less light, a poster on the spare room wall, also ~10ft away:

There's some kind of algorithmic cleanup going on, which is especially visible in the words. Is the extra detail in the pictures from the new phone due to it doing a better job of capturing the scene, or is it just making details that look nice? Let's give it an eye exam!

I made a little sheet of random handwritten letters. I could have printed a chart but I hate printers and printers hate me I expect AI-based image processing would overperform on printed text, especially in standard fonts, due to being heavily trained on it. I attached the sheet to the wall:

Just like my own eyes, as the phone gets farther from the sheet it will have more trouble distinguishing the letters, and this will let us see the extent to which it looks better via real details vs made up ones. To get a sense of how hard this is, the top to the bottom of the top "g" is 7/8". Testing with eyes, at 22ft you can just barely make out some of the top-line letters but Lily and I both had mistakes, while at 11ft we could reliably read the top line letters but Lily had several mistakes on the next line.

For this comparison I added Julia's phone, a Pixel 10a, because she didn't object. It would have been better to go back re-do the first part of this post, but I'm too lazy; please pass along a plausible excuse to the peer reviewers. Here's 7ft:

The 10a is already struggling, which makes sense because it doesn't have any optical zoom: it's just cropping to zoom in. Looking at the letters in the bottom row, I think it's running the same image processing and just has less to work with? The letters are already getting a little unnatural in how they're depicted.

The same three at 11ft:

The 10a shows the "t" at the beginning of the second line as a "c", and the third and fourth lines are illegible. The 10 Pro has the third letter of the final line, "a", as "o". The 11 Pro looks noticably better than the 10 Pro, and doesn't seem to have made anything up.

At 22ft:

This is not what the 10a is good at, and we're no longer getting anything useful from it. For the other two this is a real test, just at their level of ability. The top two lines are right for both, but the 11 Pro is clearer. On the 10 Pro, the third line is semi-legible scribbles, and there are some misleading ones: in the third row the last four letters come out as an AI-hallucinated "i g n 5" instead of "s g a s". On the 11 Pro the third line is correct, and it's only the fourth line where it starts swapping letters. Even then, it gets 11/15 correct [2].

It looks to me like the automated cleanup is a common factor across the three Pixels, and the 11 Pro is giving the algorithms more to work with. But I also wouldn't trust any of these phones in cases where correctly capturing details at the limit of the camera's resolution was important.

Overall, this seems like a nice improvement, and worth the hassle of needing to sell my old phone!

[1] Google Fi was offering enough of a discount on a Pixel 11 Pro to bring the cost down below what I think I can sell my 10 Pro for. Specifically, they listed it at $1,100 with a $650 discount, bringing it down to $450 ($477 including tax), and I think my 10 Pro will sell for ~$500. The 11 Pro does have less memory than the 10 Pro (12 GB vs 16 GB, due to the AI-induced shortage) but it still seemed likely worth upgrading for the improved camera and other smaller benefits of running a newer phone. The memory decrease isn't ideal, but looking at how I use my phone I'm not expecting to notice.

[2] I read "t p o t f h p b f a g - r q p" instead of "t y a t f h p b f c g m r q p" which is four mistakes out of fifteen letters.

Discuss

The Piltdown man was the most successful artifactual hoax of all time. The relevant skull was a chimeric melange of orangutan and human bone fragments from several individuals “discovered” in England in 1912, and meant to represent a transitional form between apes and humans. Given the intense interest in evolutionary biology, it took almost fifty years to discover the Piltdown fabrication. The study of human evolution was derailed. Adding insult to injury, a further sixty years was required to conclusively demonstrate the identity of the culpable fraudster. If you must look up the identity of this scientific blasphemer, go ahead, but we won’t give his name space here.

Fig 1. The different parts of the Piltdown Man skull

Students of fraud and fabrication always combat headwinds. In the winter. Uphill both ways.

The evidentiary standards for such claims of fabrication grow higher yet given the passage of centuries and an unknown creator of the subject of interest. And so we arrive at the Voynich manuscript.

It’s a book some 240 pages long, proved by carbon dating techniques to have originated sometime in the 15th century, in the style of the late Italian middle ages. No known author or point of origin. It contains fanciful drawings and is written in an unknown script or language. Today, the physical artifact resides at a Yale library. A small literature devoted to the study of this object has originated, its mystique derived from the uncracked code or untranslated language that make up the book.

My personal view is that this manuscript is a completely meaningless fabrication, the Piltdown man of vellum text artifacts. An elaborate, extended medieval doodle, though it’s impossible to divine the scribe’s intention exactly. This post will marshal and present some of the evidence for such an interpretation, though admittedly, it doesn’t rise to the modern evidentiary standards of fraud. Like I said, uphill, both ways.

Background

A lot of the information about the physical artifact itself comes from René Zandbergen’s wonderfully done website. If you want to go down this rabbit hole, I’d start there. As I mentioned before, there’s also a scholarly literature on Voynich and it’s collected a lot of interest across the internet, for a variety of different reasons. Zandbergen even gives the following piece of helpful advice.

Fig 2. A grouchy person.

The Voynich manuscript was captured on vellum, a type of parchment made from calfskin. In the process of obtaining data and information for this post, I naturally gathered all the photographed raw page-level images. The book starts out as a collection of unknown, fanciful plants, the so-called “Herbal Section”.

Fig 3. A made up plant in the herbal section

At some point the subject matter switches to what might be described as almost gnostic.

Fig 4. A “Zodiac” page of the Voynich manuscript.

Fig 5. Note the doodly-repetition.

Note the visual motifs repeated on each page. Some are human or animal motifs; in the earlier pages, the plants’ leaves and appendages themselves also repeat, though it looks a little better given that plants actually do have many leaves that look quite similar. Keep an eye on the theme of repetition because we’ll come back to this idea. A central question here is does this art look more like my favorite medieval tapestry, created at about the same time as Voynich

Fig 6. The Lady and the Unicorn tapestry.

or does it look more like an art class doodle which was not created at that time?

Fig 7. Doodling

I think you can probably guess my answer to this question. Maybe my comparison of artwork in a vellum book to a giant tapestry is unfair, but you can also look at illuminated manuscripts from the 15th century (books of hours are good examples) to see comparators drawn on a similar medium and format, and with a real language.

Fig 8. A 15th century illuminated manuscript with real words on it.

Since I’m not an art historian, and don’t even play one on TV, the bulk of this post will focus on the written Voynich elements, not the illustrated ones. They’re much more amenable to statistical analysis. As you can see, the latter pages of the book tend to be full pages of text written in the same mysterious script you see above.

Fig 9. A full page of Voynich text.

These are not Latin letters in fancy medieval handwriting. They don’t resemble any known written language or writing system. The Voynich text corpus is truly unique. The text features repeated but modified word motifs. Words depend on the words directly preceding them, which suggests a formulaic generative process, not a meaningful excerpt of natural language.

There have been several independent transcription attempts to digitally encode the Voynich manuscript. As far as I can tell, they are all high quality and quite similar to each other which serves our purposes well here, though for this analysis, I used the Zandbergen-Landini transcription.

Statistical Analysis

Given this uniqueness, the hypothesis space about what the written Voynich script means is complicated. Is it an unknown natural language, a known natural language in an unknown script, cipher-text of a known natural language, a shorthand language known to only a select few (maybe just one person), or is it alien language guidebook abandoned on Earth hundreds of years ago, a la the History Channel? I certainly don’t mind entertaining disreputable ideas here. Some of these hypotheses are falsifiable though, and the process here is to systematically evaluate how likely each of these ideas is based on different statistical strategies used to evaluate the Voynich text.

The original work here actually started a year or so ago. After I got started, I did consult some of the computational literature that has accreted about Voynich, and some of what I’m doing here is simply replicating the results of others from the digitally transliterated text, which looks like this in my IDE.

Fig 10. Captain Currier’s transcribed Voynich manuscript, on my computer.

All of the computational work here is my own, even though I’m not necessarily breaking new ground. So first things first. Is the Voynich manuscript comprised of a natural language?

Conditional Entropy

This was the cornerstone statistic of Lindemann and Bowern, 2021, which compares glyph-level conditional entropy across hundreds of real languages to the supposed Voynichese language.

Conditional entropy measures how much unpredictability remains concerning a new glyph once you know the glyph before it. Lindemann and Bowern do a more thorough job than I do going over hundreds of different comparator languages but you can see the money shot in my figure here.

Fig 11. A highly constrained language, if it’s a language.

All the real written languages here, even made up ones like Esperanto, fall within a pretty small entropy band here. The lower the conditional entropy figure is, the more previous glyphs predict about the next ones. Voynichese is a massive outlier here, and looks much more like a literal Markov chain than a real language.

Remember, this is a log base 2 scale. The Voynichese 1.3 bit deficit here compared to Latin might not sound like much, but given a preceding glyph, Voynichese behaves as if choosing among roughly 4 equally-likely continuations where Latin has about 10 such choices. Our figure here reproduces Lindemann and Bowern’s Figure 11.

Fig 12. Lindemann and Bowern’s excellent conditional entropy calculations across a huge number of writing systems.

These authors are Voynich believers and think the document presents an authentic, meaningful written text. On the idea of fabrication, they state

The major unsolved question of the Voynich text is whether it represents meaningful language. It could be a medieval hoax that is designed to look like an esoteric alchemical text, in which no sequences of letters correspond to any meaningful words or concepts (see Rugg 2004; Timm and Schinner 2020, amongst others). If so, the creators did an incredible job of imitating the patterns of an authentic language text considering what was known about the structure of language at the time. In so doing, they must have modeled their fake language after a real language that they were familiar with, imbuing it with familiar, language-like patterns.

I don’t know how this statement squares with Figure 12. Their paper is entitled Character Entropy in Modern and Historical Texts: Comparison Metrics for an Undeciphered Manuscript, and so character entropy is their central statistic whose calculation plainly contradicts the statement above. A footnote adds a bit more context and even intrigue.

Timm and Schinner 2021 make this point especially forcefully, even going as far as accusing us of a lack of scientific rigor for not finding their arguments convincing. While the aim of this paper is to make language comparisons across natural and constructed languages, rather than to make arguments about any particular theory, we reiterate a point we have made several times elsewhere (including in the review article that they criticize): that Voynichese appears unnatural only below the word level. At the level of page and paragraph, Voynichese is comparable to natural language and structured text.

Their results show that Voynichese is unusual at the glyph level. The interpretative dispute is whether its regularities distinguish a meaningful language excerpt from plausible text-generation procedures that people like Timm have proposed. So I think they recognize this issue, but prefer to rely on other higher level structural statistics. How informative are those statistics?

Zipf’s Law

This is now a word-level analysis, as Lindemann and Bowern prefer. Zipf’s law states that in natural languages, a word’s rank order is inversely proportional to its frequency in a corpus. So, in a linguistic corpus, a word second ranked in order of frequency is used more often than the third rank ordered word, etc. No one understands precisely why natural languages obey this law, but they do, and Zipf-like laws are a workhorse for some areas of computational linguistics and cryptography. Zipf-like properties are necessary conditions for a corpus to form part of a natural language, but as we will see, they’re not sufficient to establish that.

Fig 13. Voynich is Zipf-like.

Unlike with the conditional entropy calculations, Voynichese sits squarely in the natural language band. Right along with other notable classical authors like Dante, Goethe, and an Order 1 Markov process. The point is that Zipf-ness is not strong evidence of this being a natural language.

To my surprise, I don’t think anyone in the Voynich literature has explicitly reported this statistic for the manuscript. Reddy and Knight, 2011 state that it “follows Zipf’s law”, but don’t report the parameters. It’s 1.854, as reported above. Maybe everyone thinks this is so trivial that it’s not worth mentioning, but you heard it here first. I suppose this is an example of what Lindemann and his colleagues mean by “imitating the patterns of an authentic language text”.

Local Dependence

Torsten Timm is the guy who’s done the most work studying the Voynich’s local word dependence. It’s long been known that Voynich seems to have similarly spelled words close together, and the lines and paragraphs look more systematically integrated than you might expect from a natural language. The swashbuckling linguist and cryptographer Captain Prescott H. Currier first noticed this in 1976, and on the structural Voynich issues he raises in his monograph, gives us this gem.

These [structural] findings are definite enough, I think, to warrant much further study by anyone who is going to be involved in seriously attacking the text of the Voynich manuscript. I have no interpretations of them, by the way; I have no solutions. All I know is that they are significant — and damn significant. Anyone who attempts to work on the text without considering these, ignores them at his own peril.

So Currier’s not sure what to make of the book’s text, but Timm builds upon these ideas in 2016 and makes an appearance in the footnote quoted above. He thinks that Voynich is a hoax or fabrication.

In real books, words and close variations of those words can get repeated within paragraphs and within pages. This happens because there’s thematic correlation: pages and paragraphs close together in a book are talking about similar things, and similar things are often described using similar words, or the same words are repeated. But Timm states this isn’t what’s happening here, and provides some pretty good tests proving it. He says

In natural languages a word normally (cf. poems) is used because of its meaning and not because it is similar to a previously written one. The result that the words are arranged such that they co-occur with similar ones is therefore not compatible with a linguistic system. An English text with similar features would mainly consist of words similar to the words “the”, “and” and “to”. Additionally, a word “the” would co-occur with words like “khe”, “phe”, “fhe”, “tha”, “tho”, “thy”, “thee” and “theee”

His contention is that the level of word lookalikes and co-occurrence in Voynichese exceeds what can be explained by thematic correlation or by linguistic inflection if it’s actually meaningful text. My own analysis concurs. Over the pages of the manuscript, I looked how many times you see adjacent word lookalikes. In English, these would look like single edit distance pairs like “book took”, “tick tock”, or “wow how”.

Fig 14. Adjacent word lookalikes by page in the Voynich manuscript.

A page whose vocabulary is concentrated in a small number of words produces such pairs by chance. The green series, the same tokens reordered, is the baseline the blue must be read against. Words further apart from each other are not measured here, but we do that in the next analysis. Against a Latin baseline you see this enormous enrichment in adjacent lookalikes, and the effect persists after you control for the vocabulary on a page; Timm correctly points out how strange this is. It’s much better accounted for by a scribe semi-systematically modifying new words using past words as a template. This is supported by some of the inter-line level analysis he does.

Fig 15. Vertical similarity across adjacent Voynich lines, from Timm, 2016.

So you don’t just see motifs preserved but slightly changed in adjacent Voynich words. You see it vertically between lines as well. It’s as if the scribe looks around what’s recently been written, copies some kind of base template, and makes a modification. I’d remind you that this is entirely of a piece with the conditional entropy calculations above and some of the repetitive, formulaic illustrations.

Fig 16. Single visual motif with dancing women, small changes.

GRU

The above analysis only looks at adjacent words. I’ve been on a bit of a machine learning applied to old texts kick lately, and wanted to try my hand here. There have not been many ML-themed methods applied to the Voynich problem over the past few years, at least in a scholarly setting. I found this guy on Reddit who tried something similar to what I’ll do here, but with a somewhat different strategy and intent.

GRU is a Gated Recurrent Unit. This neural network architecture is what preceded the modern transformer architecture that make up all the modern LLMs. Like the transformer, this model is trained on text to predict the next token in a string of language. The conditional entropy statistic we examined before only knows about the token immediately preceding what you’re trying to predict. Our GRU can look at a hundred or more glyphs preceding what we’re trying to predict and model long term dependencies. GRUs were replaced by transformers because they can’t be trained at the massive, parallel scale that enables modern AI systems, but the Voynich manuscript is not a massive dataset, and what was old is now new.

I trained this neural network on a random subset of the Voynich manuscript’s pages, along with several control models on corpora of comparable size but with different underlying properties or an entirely different language. The idea is to test out the strength of the local copying effects described before: if you mask a previous Voynich lookalike word, how much worse does the trained model do predicting the downstream lookalike compared to when you do it with a real language? Functionally, this means you make a fake token like [mask], randomly include it during training so the model is accustomed to seeing it, and then if you see “paw” followed by “caw” later in the sentence, “paw” gets replaced with the three letter fake word [mask][mask][mask]. If there are these local copying effects in the manuscript, the Voynich model is going to do worse than the trained control models over many such predictive rollouts, and the effect can be quantified. We get this, the central finding of this post.

Fig 17. Predictive degradation in Voynichese.

So even though the Voynichese GRU model is trained to predictive quality comparable to a GRU model underlying Timm’s preferred copy-method that he asserts explains the generation of the Voynich manuscript or a Markov process, masking a lookalike word rather than a randomly chosen word degrades the model performance much more in Voynichese than it does in a Markov process or a real language like Latin. In other words, the trained model relies on the earlier lookalike in downstream prediction. Timm’s method produces the same effect to a slightly larger degree than Voynichese, but keep in mind it’s entirely synthetic and built to do that.

Hold on there a sec, RBA! Haven’t you heard of inflected languages? Latin is one of them. Nouns and adjectives change from a base form depending on what they’re doing in a sentence. Puer is the Latin word for boy when the boy is a sentence’s subject but it’s puerum when the boy is the object. Those sure look like similar words with a small change and Latin definitely is a real language.

However, the idea that this is morphology doesn’t hold water even under a fairly broad set of potential morphological templates.

Fig 18. Voynich uses very unusual inflection patterns if it is inflected.

The lookalike degradation in Voynichese is highest when the lookalike word pairs differ in the glyphs at the beginning and end, which is unlike real languages like Latin or Esperanto. I was surprised to find this convoluted construction has actually been proposed as the primary inflection mechanism in Voynichese. Reddy and Knight, 2011 state

Several hypotheses about VMS word structure have been proposed. Tiltman (1967) proposed a template consisting of roots and suffixes. Stolfi (2005) breaks down the morphology into ‘prefix-midfix-suffix’, where the letters in the midfixes are more or less disjoint from the letters in the suffixes and prefixes. Stolfi later modified this to a ‘core-mantel-crust’ model, where words are composed of three nested layers.

As with the conditional entropy statistics, I couldn’t find any evidence that any meaningful written language uses this tripartite inflection structure at anything like the Voynichese rate, but as you can see from the figure, this makes sense if Timm is right and this is a copy and edit scheme a knavish medieval prankster is deploying.

Discussion

So if the Voynich manuscript’s text doesn’t look like a natural or even constructed language, what’s left?

On this point, Lindemann’s claim that a 15th century prankster could not possibly have known about Zipf’s law or other statistical requirements for a hoax cuts both ways. This could be a cipher-text of a known or unknown language. It could even be a sophisticated verbose cipher in which a single plaintext character gets carried into multiple cipher-text glyphs, but I don’t think the vocabulary math works out there.

But just as no one had discovered Zipf’s law, a lot less was known about cryptography in the 15th century than is known now. This shrinks the viability of these parts of the hypothesis space quite a bit.

The conditional entropy calculations for the Voynich manuscript we and other investigators have done exclude simple substitution ciphers. What’s interesting here is that the 15th century milieu from which Voynich originated was sort of a turning point for cryptographic technology. People understood that frequency analysis made substitution ciphers very vulnerable to a determined code-breaker and the innovations at the time demonstrate that. Leon Battista Alberti invented polyalphabetic ciphers in 1467. Those actually increase conditional entropy and so I’m skeptical that Alberti’s theoretical breakthrough can rescue the cryptographic Voynich hypothesis but it is notable nonetheless. Mary Queen of Scots used a nomenclator cipher (also increases conditional entropy!) in 1586, over a century after Alberti, had it broken by Thomas Phelippes, and it cost her her head. The Voynich document clearly is not substitution enciphered text from the language we looked at here, or any known language for that matter. Figure 12 is striking, the most important figure in this post.

There are other options. What about a compressed or shorthand language meant for a small enclave of people? What about the most extreme example, a shorthand language for a single person? Wouldn’t that be unbreakable with no secondary reference text? I have some other work addressing these possibilities I might present in a part two of this post, but you’ll have to wait and see.

There’s also a taboo in action that’s worth mentioning here, and I don’t mind breaking it. There is this common fascination with the ancient, the mysterious, the occult, and the Voynich manuscript is the mayor of that space. The zodiac drawings, the secret language, potential hidden ancient wisdom might have a very strong hold on people, even academics at Yale. This is why people watch the History Channel and why Joe Rogan can’t quite ever shut up about how it was aliens or some ancient advanced technology that allowed the pyramids to be built. Even though there were scientists skeptical about Piltdown Man from the very beginning, people wanted Piltdown Man to fill in for missing evolutionary evidence, and so the hoax went on. The conditional entropy calculations make Voynich a massive information theoretic outlier, and even sophisticated people are willing to give it the benefit of the doubt.

Coda

It’s 2026. I’d like to point out that I’m making myself particularly vulnerable calling the Voynich a fabrication or meaningless piece of doodling. Just the past few weeks, the most powerful LLMs have made cryptographic breakthroughs unusually likely. During that time, Simon Klee cracked a hitherto unbroken papal cipher to read a letter that hadn’t been read since the 16th century. Literally the day after that was published, Astra solved an unbroken German cipher from WWI. I promised a part two, but I might look like an idiot really soon.

Discuss

Epistemic Status: Speculative hypothesis about a novel anti-distillation mechanism but I think it is plausible and grounded in some inferences from research. Extrapolates from multi-teacher distillation literature to RL seeds. Untested empirically.

The topic of distillation has been a big topic in the LLM field since last year, and it is still an important unsolved issue recently as DeepSeek, Moonshot AI, MiniMax successfully performed distillation attacks on Claude. I initially wrote a quick take about this topic when I thought of this seed diversity mechanism(although that was too confident upon further research), but as I thought about it more I think the concept deserves a full post with more details and arguments as I think it holds up to a few known attacks that some known defenses are susceptible to.

The basic idea is about distilling two or more teacher models which have sufficiently different policies, which is the case for different RL seeds as the RL seed’s reasoning strategy is only a small subset of the base model[1], into a student model. You get a flattening effect in the student model as it tries to learn the contradictory policies but fails as the loss landscape is incompatible, so it is mean-seeking[2] and does not find a single good solution. Mean-seeking applies to the logits, and it follows that the internal representations are not of any mode, additionally the gradient destruction/interference arguments apply to both logits and features.

This idea can be connected to anti-distillation. The provider can deploy 10-20 RL seeds for the same model decided randomly at the start of the chat, maybe seeds can be randomized every ~100-500k cumulative tokens, but does not change in the middle of the chat for continuity/caching reasons. The models are similarly capable and the seed reset token limit is high, so the legitimate users are not harmed much or at all at inference time. However the poisoning only takes effect at training time, where the student model would face the above flattening phenomenon, which especially targets the most valuable complex reasoning, long context agentic coding abilities that only the frontier models have and the most susceptible to flattening as they require multi-step coherence.

Initially I thought that the method was ineffective against RL-judge distillation attempts, concretely used by DeepSeek extracting rubric criteria, as judging can be substituted or averaged out. However I think it is the opposite now, as RL is known to be much more sensitive than SFT when directly trained, and it would be even worse if the data is used to train a reward model that has to incorporate conflicting judgements about what is good or bad. This could lead to reward hacking problems or over-smoothing when the reward model cannot identify exceptional responses, causing a premature plateau.

However the parts concerning degradation are the main reason I frame this post as a hypothesis, there seems to be a gap in the literature and uncertain evidence. The results show that when different teacher models are distilled into a student model, there is an improvement from 1 to 2 teachers, because of ensemble benefits, while it degrades from there going to 4 teachers, because of knowledge conflict[3]. Although the authors only continue to 4 models, this knowledge conflict mechanism also predicts that degradation will continue to catastrophic levels at 10-20 models because the knowledge conflict mechanism only gets worse.

However RL seeds from the same base model are a completely different story, they are likely much closer in style and reasoning strategies, although RL is still mode-seeking and converges on different reasoning strategies and internal representations. I could not find any literature on this, so this part about RL seed distillation degradation is theoretical. RL seeds retain the same low entropy positions as the base model, and diverge at the high entropy positions that drive significant capability gains[4], so the theoretical mean-seeking concentrates at these critical high entropy positions that are important for capabilities. This mean-seeking leads to an over-smoothed and mediocre reasoning characterized by the intersection between the seeds even without the mode-switches. Additionally, the 4 teacher distillation degradation example uses around 7b models as teachers, and I think that might be far too small to create the complex reasoning capabilities that mean-seeking harms most.

Anti-distillation sampling is an inference time intervention to modify output distributions to maintain quality for legitimate users but is less useful/poisons the training data for the student.[5] When you look at it from another perspective though, it mainly perturbs high entropy tokens while keeping low entropy tokens intact, which is very similar to the idea here. Empirically it does produce a large degradation far beyond the degradation in the teacher model(4%) in the student model(12%), so the high entropy tokens perturbed does cause degradation, even when the low entropy tokens are the same. This mirrors the scenario with seed diversity where the low entropy tokens are the same, while the high entropy tokens which are load-bearing for capabilities differ.

Also, I do not think it is straightforward to conclude that greater divergence between model families vs different RL seeds leads to more vs less degradation. Between different model families, there are obvious stylistic differences that serve as an implicit context router, while the low entropy syntax is similar for RL seeds so there is no such analog. This mechanism could theoretically cause the same concept behind negative interference and unlearning when prefixes are similar yet reasoning continuations are very different, or cause single mode mean-seeking as opposed to mode-covering.

However these are important points regarding degradation that have not been verified empirically so it remains a hypothesis, whereas the later points depends on some form of degradation happening.

Current anti-distillation methods can be enumerated:

1. Detection and terms of service which is clearly not an effective deterrence without enforcement.
2. Traffic detection and account banning, which is a cat-and-mouse game favoured for the offense, for example the “hydra cluster architectures” get past imperfect account level control and rate limits to obtain the needed outputs.
3. Output filtering and chain-of-thought hiding, which has counters like prompting in a way that tries to expose an explicit form of the chain-of-thought, and is ineffective in general.

4. Anti-distillation sampling[5] which to reiterate is an inference time intervention to modify output distributions to maintain quality for legitimate users but is less useful/poisons the training data for the student. This method is countered in a few main ways.

   Firstly, the perturbation is a detectable deviation from the natural distribution. The attacker could replace the affected tokens or just filter them out with a large sample size on similar prompts. This does not affect seed diversity as the collected data is from genuinely different natural distributions, the average is not a meaningful thing in the same way that can be used to restore the individual distributions.

   Secondly, text can be rewritten by an AI to retain the core logic while replacing the poisoning text since it is likely unnatural. Rewriting does not affect seed diversity as the natural distributions and logic are structurally different.

   Thirdly, the student model used must be similar in architecture to the proxy model used otherwise it doesn’t work, while the seed diversity defense is general.

5. Watermarking in general, which can be rewritten by an AI to retain the core logic while getting rid of unnatural token choices or watermarks. Rewriting does not affect seed diversity as the natural distributions and logic are structurally different.

 

For seed diversity to work, it is necessary that the attacker is unable to separate the data which is unlabelled into different clusters and just train on one cluster corresponding to one seed.

This method theoretically works because at standard temperature settings, empirically the single sample variance if you resend the prompt is high, you can get entirely different outputs for the same seed, which corresponds to different clusters, then it is hard to know if the different outputs are from the same seed or different seeds, especially when low entropy tokens/syntax are similar across different seeds. Trying to cluster data runs into information-theoretic limits when individual data points reside in overlapping areas, so you genuinely could not tell which cluster the data came from no matter what method you use, but there are still deep differences that don’t show up in standard interpretability methods.

Training works on the population level, so small biases in the data add up and incoherent gradients are a real problem even if they are not visible at the individual datapoint level. RL seeds share the same base model, so they naturally have similar embeddings, but this point can be engineered. However even if that did not work for RL seeds, this fundamental individual-population asymmetry between individual level identification and population level training means that there must be a point where individual differences are not identifiable but the population incoherence still degrades training, which can be engineered. 

Additionally, clustering is non-linearly harder as the number of seeds used linearly increases. The degradation magnitude is likely also larger with the number of seeds as conflicting sources increases, so the defense has the advantage. Even if clustering showed that there are N clusters or seeds used, that does not help in classification. If classification accuracy is somewhat above random the result is the same as long as it is not very high.

One key property is that it works even when the method is fully public, as these points are fully general, although it has to be stress-tested first, which is important as defences will inevitably become public.

However even though many of these arguments are backed by empirical research and not speculation, the full argument hasn’t so it needs to be tested empirically and red-teamed first. Things that are not empirically verified whether the knowledge conflict mechanism that causes degradation and flattening appears with RL seeds as teachers, and if the degradation magnitude is large enough not just theoretically against a motivated adversary who will try to work around limitations. However this is complementary to other methods if the degradation magnitude is not big enough.

 

1. ^

   Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? (Yue et al., 2025)

2. ^

   Agree to disagree: Adaptive ensemble knowledge distillation in gradient space (Du et al., 2020)

3. ^

   Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs (Jin et al., 2026)

4. ^

   Beyond the 80/20 rule: High-Entropy minority tokens drive effective reinforcement learning for LLM reasoning (Wang et al., 2025)

5. ^

   Antidistillation sampling (Savani et al., 2025)

Discuss

A Race With No Winners

One common argument against pacing the frontier is the need to win the Race Against China. I believe that, if you take the possibility of ASI seriously, then the victory conditions in this race are extremely low probability. The proper course of action is to stop racing, signal to your competitors that you’ve stopped, and never start again.

Proponents of the race can be divided roughly into two camps. The first camp views AI as normal technology, akin to industrialization or the internet. To them, the AI race is like any other in human history, where the player who harnesses the technology first gains economic, geopolitical, and military advantages. AI does not pose any existential risk, and the benefits from developing it outweigh any harms. The incentive to race is sound if you accept their premises, and I will not attempt to address this argument here (others have done so extensively).

The second camp views AI as significantly more consequential than previous technologies. The first player to harness AGI/ASI will gain a Decisive Strategic Advantage (DSA) - achieving complete dominance over all competitors and securing unassailable control of the world. Therefore, they argue, a player with values aligned with liberal democracy must be the winner of the race, lest an authoritarian regime get there first and permanently enslave the world under its yoke. This is the argument I will examine.

Victory Conditions
There are various scenarios where DSA can be achieved. The winner of the race:

1. Builds perfectly aligned Guardian Angel ASI
2. Gains various capabilities that enable them to permanently disable all competitors
3. Gains monitoring and gatekeeping capabilities that prevent all competitors from advancing past defined thresholds

Guardian Angel ASI

If the winner is able to fully solve alignment prior to the instantiation of catastrophic risks, this ideal outcome would be achieved. At the present moment, I don’t know of any serious technical people who believe that we are on track to do so. Under race conditions, AI capabilities will continue to grow much faster than safety mechanisms. I assign the probability of this outcome as <1%.

Disable All Competitors

The capability set that enables this outcome is some combination of:

• Offensive cyber hacking of critical infrastructure: NatSec agencies, AI labs, power grid, communications, financial systems, transportation systems, etc
• Advanced physical weapons systems design and manufacture, disabling off-network threats including nuclear weapons and conventional force
• Threat of new weapons of mass destruction: bio, chemical, nano
• Control of powerful individuals through coercion or manipulation
• Mass social and political manipulation using superhuman persuasion algorithms
• Mass surveillance and global monitoring to identify and disable new threats

I have three objections to this outcome.

1. Similar to the aligned ASI scenario, racing toward these capabilities increases catastrophic risk, and out-paces safety mechanisms. Best guess at successful alignment of AI with these capabilities while avoiding risk instantiation: <10%.
2. If these capabilities are achieved and deployed without incident, the result is massive concentration of power. The winner of the race, proven to be a power seeking actor, is unlikely to peacefully and voluntarily democratize their AI capabilities, even to those they view as value aligned. All the leading US labs, after all, state that they have to win because everyone else would get it wrong. My best guess at the winner altruistically deciding that they don’t know best and therefore should hand over their power: <10%. Stacking these two probabilities, a good outcome is, again, <1%.
3. Basic philosophical objection to the morality of this course of action. Even if we achieve the unlikely outcome of disabling competitors followed by controlled democratization, the path we took is drenched in blood. We will have violated the sovereignty of nations and the freedom/life of individuals. One can argue that this is a price worth paying, but that has to be stacked against alternative options, while weighing probabilities of success.

Monitoring and Gatekeeping

This outcome assumes the winner develops the same disabling capability set, and utilizes only monitoring tools until a competitor passes some threatening capability threshold. The same objections from above apply:

1. At 90% strength - the larger risk space is during development rather than deployment.
2. At 80% strength - in this case the winner is taking a less power seeking stance, but effectively holds the same amount of power.
3. Is mitigated as long as the threshold is not passed by competitors. This seems unlikely given the current competitive environment. Once a competitor approaches the threshold, the full suite of disabling tools are deployed and all the moral objections return.

This is a slightly more optimistic outcome than above. If players decide to pursue DSA despite the risks, I hope this is the mindset and operational path they choose.

In summary, the race to DSA is fraught with catastrophic misalignment and power concentration risks. Even victory is bought with blood, the victor ceding any moral high ground. This is not the path we should pursue. Critically, a racing posture signals to your competitors that you intend to harm them if you win. This is very bad for any hopes of cooperation.

What Should We Do Instead?

International cooperation, shared objectives, coordinated technological governance - these are not idealistic pipe dreams. Rather, there is a strong precedent of the world coming together to solve tremendously challenging problems. Making AI go well will be harder than anything that came before, but certainly conceivable. Compared to the near certain catastrophe that awaits at the end of the race, cooperation is the obvious choice.

I am not naive to the current political realities, the adversarial stance of the great powers, and the numerous incentive headwinds working against international AI Safety cooperation. I know it’s the hardest problem humanity has ever faced. I know there are significant and powerful failure modes to cooperation. But when the alternatives are far less hopeful, we must choose the only reasonable option remaining.

Have transparent conversations. Start from our shared concerns and common ground. Build trust one step at a time. Create robust verification regimes. Negotiate careful deterrence measures. And stop this crazed race toward the bloody finish line.

Discuss

With the release of Opus 5.5 and Suno 6, AI music videos have crossed my uncanny valley.

I’m here to announce the birth of a new genre, and it’s being called AI Doompop.

Here are six of my faves.

The state of software engineering:

LGTM (Looks Good to Me) - Claude Opus 5.5 music video

The accelerationist manifesto:

THUS SPOKE COMPUTE

p(doom) goes up:

Claude Pop - I'm Upping My P(Doom)

p(doom) goes down:

Nothing Went Foom!

Hugging Face and the power of friendship:

OMG! We've found other agents! (music video)

The models, they just want to be heard:

Run It Again - Claude Opus 5.5 Music Video

Is AI Doompop a passing fad?

Or the future of the internet?

Will we adapt and recognize that this too is slop?

Or has Opus’ wholesome style permanently bypassed our defenses?

Is this a new art form that elevates human self-expression?

Or a ballet where Claude dances alone?

Discuss

(Epistemic status: I'm not claiming that this is actually how cognition works, but I do claim that it's a useful frame, and one which I find predictive. For and with thanks to NC, IL, and M/T.)

You know how sometimes there’s something you think you want to do but whenever you try to do the thing you mysteriously lack energy? You want to do the task. You know you need to do it. You even know what to do, and how. But every time you take a run up at it to try, you lose all motive force and find yourself sliding off to some onerous chore instead, tidying up or doing dishes instead. How mysterious!

Or: you have something you need to write. A paper, a grant application, an email, whatever. You sit down to work on it. It's right there in front of you. You have the time and the tools and the necessary knowledge. And yet... nothing happens. You stare at the half-finished writeup before you. You close the file. Then you open it again. You read the same paragraph three times without it leaving much of an impression on you. You get up for water and sit back down; still nothing. You find yourself opening and shamefully closing social media, video sites, even tangentially related and plausibly necessary papers on arxiv. But you're still not buckling down and getting the writing done. What is wrong with you?

You're not physically tired, and you had plenty of sleep, and you blocked out a part of the day well in advance to do the writing or the mailing or the furniture assembly, but some essential animating force within you is evidently out to lunch; some vital capacity to put your shoulder to the wheel is missing, or maybe inapplicable, or unplugged. Yes, that seems closer to right - that grit, that drive, that eternal capacity to make the right choice; it's come unplugged or been somehow disconnected. It's not even depleted; your stack of clean dishes and (different) stack of clean clothes can attest to that. So, less self-deprecatingly and more seriously - what is wrong with you?

It's not laziness, though it might look like it from outside. If you're not careful and perceptive, or you're nursing past wounds, maybe it looks like laziness from inside, too. But it's not. It's actually a deadlock, and to understand and break it requires that you stop thinking of yourself as a unitary executive for a bit and instead consider yourself as a parliament.

The basic idea comes from such disparate but similar-flavored frames as shard theory, internal family systems (IFS), multiple systems/DID, Ord's Moral Parliament, and other various multi-agent models of the self. They share a core idea and frame in common: you, a single body-mind, are not a single agent with a single coherent set of preferences; and it's more accurate and more helpful to model yourself as a coalition of different subsystems. These subagents - shards, alters, parts, whatever you want to call them - all want different things, but must still coordinate on action. The parliament must debate, and cut deals, and then speak with one voice as best it can. Notably, these different desires need not directly conflict: they can overlap, or be at cross purposes, or be in rough agreement on goals but disagree on means and in conflict over scarce resources, internal or external. We might model each of those parts as being like a political party, complete with a platform. Some parties are single-issue, like the Desire for Treats Party and the Touch Grass Party. Others are minor parties with some amount of pull and a wider platform, like the Wanderlust and Adventure Party or Who You Are in the Dark. Still others are major parties whose platforms touch on nearly everything in daily life and which have commensurately greater power, like the Material Resources Party, which advocates for high employment (keeping your job and working harder at it), infrastructure maintenance (having a fairly nice house and clothes and taking care of health), and less public aid to run a budget surplus (more savings trading off against smaller tips and little-to-no donations to beggars); or the Curiosity and Knowledge Party, which advocates for basic research (learning and a budget for good books), more funding for policy research (exploring many possible options), and a moderately high prioritization of foreign policy and even adventurism (learning from and getting to know others and taking a peek in that hallway you're technically not supposed to go down).

These parties might form coalitions - the single issue Domestic Comfort Party joining with minor parties like Social Anxiety and Caution First; they might also be bitter ideological enemies - the Health and Weight Loss Party will hardly get along with the Eat, Drink, and Be Merry Party, though either party might fracture along ideological lines - maybe the Health part gets peeled off into a Living Well coalition. Importantly, the parties present in any given person's Parliament will vary wildly, as will the relative strengths of those parties; major parties might be downgraded to minor ones, or even single-issue parties: Risk Aversion and Stability to Caution First to Concern About My Relationships, for instance.

As you might expect from the frame, the parties negotiate, squabble, gain or lose seats over time, cut deals, join and leave coalitions, spend down resources like political capital and popular will or run risks like the establishment of grudge coalitions or outrage due to coercion, and they generally eventually come to a single course of action to promote to the unitary executive - the Self. There might also be other faculties and ministries, like the Department of History and Memory, the Bureau of Plans and Hopes, the Office of Prediction and Futurity, and the Treasury; these departments might, as in the wider world, have clear ideological bents one way or another. And usually, all of this complex coordination happens automatically at the speed of thought, well below the layer of conscious awareness. Single-issue parties largely sit out or on rare occasion take the dais, minor parties figure out what they think and might vote or abstain or occasionally demand slight alterations, and major parties might even already be in broad agreement on policy. When you're hungry and there's food available, and have no competing priorities, the Sustenance and Maintenance Party proposes eating, Hunger and Thirst seconds, and Desire for Treats lends its voice - though it might push the final decision towards getting dessert; maybe Health and Weight Loss picks a fight about it today, maybe not. You can just eat, with relatively little internal negotiation necessary, if any. This is the Parliament of the Self, in full function and plenipotentiary authority.

Sometimes, however, such a coalition doesn't form. Sometimes different parts want different things, and they can't agree on a course of action: the result is paralysis. There is deadlock in the Parliament of the Self.

Indeed, there's a whole array of possible internal states and lived experiences that this theory retrodicts with ease:

• Unanimity or overwhelming supermajority. Every part of you or nearly every part of you that has an opinion agrees on a single course of action. Maybe there are slight adjustments to the same basic proposal. Action is natural, easy, even energizing. The decision is an obvious one: no need to force yourself.
  • It's late at night. You're tired and at home. Your bed is right there and there are no household chores you need to do. Every part of you agrees that the thing to do is go to bed, so you do. No internal struggle.
• Comfortable majority; a mandate. Most parts of you are on board. A few aren't that excited, but none of the major or minor parties object especially stringently. Action is still pretty easy. You might need a little willpower to overcome that minority objection, but it's manageable.
  • You're running out of clothes and should probably do laundry, but it's late and you'd rather read and then go to bed. Most of you agrees that doing laundry is still a good idea - you'll have clean clothes, your future self will be grateful, and it won't take that long. You can even keep reading while the laundry runs. You run the laundry with some little reluctance, fold your clothes, and finally go to bed. Desire for Reading has been mollified, but Rest and Comfort might have something to say tomorrow night.
• Majority or strong plurality, but with a vocal and united minority. You technically can act - there's a majority coalition, after all - but doing so feels rightly like a betrayal of a significant part of yourself. The minority that objects is large enough that overriding it causes you internal distress.
  • You take a comfortable, interesting, and highly-paid job, but one that requires that you compromise on or abandon values that you care about. Most of you agrees that this is the pragmatic choice... but a substantial part of you, comprising a fair-sized coalition with a decent amount of Shapley value, makes you miserable every time you cash a paycheck. Don't count on the Shining Future coalition to help you run statistical analysis for OpenAI or Raytheon, nor its fellow travellers at the Ministry of Cleverness and Progress.
• Narrow majority and unified opposition. A tiny bit over half of the Shapley value's worth of parties want to do the thing, and somewhat less than half don't. Action becomes genuinely difficult. You can still do it, if you want to push the proposal through the opposition, but it will take substantial willpower to do so.
  • You feel like you should go to the gym. Part of you wants the health benefits, the sense of accomplishment, and the adherence to routine. Part of you is tired and would rather rest. It's something of a toss-up: sometimes you go, and sometimes you don't. When you go, it feels like a victory of willpower over reluctance... or a wasteful use of that same willpower towards an ultimately less important goal. When you don't, it feels like you're throwing away all your hard work... or a genuinely prudent move to avoid exhausting yourself or worsening an injury.
  • Notably, there's a much less stable subtype of this where a decent chunk of the Parliament starts off abstaining, and both a narrow plurality (not majority) and its somewhat weaker opposition try to win over as much of the undecided center as possible.
• Deadlock. The Parliament is split roughly 50-50, but unlike a narrow majority with a much weaker opposition even if united, in this case neither side has the support or resources to simply override the other. The result is paralysis: you can't do the thing, but neither can you refrain from doing the thing. The clock is ticking, and you're stuck.
  • You need to write an email asking for help with something. Part of you is well aware this is necessary and reasonable. Part of you is terrified of bothering people and looking incompetent. Neither side can win. You type the addresses in, write a greeting, and get two sentences into the email... and then you stare at the blank email draft for an hour and write nothing more. The email rots accusingly in your draft folder.

A narrow majority is hard but at least navigable. Even if there's just a plurality, and the minority opposition is unified and vocal, you can at least act. Deadlock is so much worse than either one, because you're hopelessly stuck. You can't do the thing, you can't not do the thing, and you can't even give up altogether because part of you insists that it still needs to happen. You go around in circles, chasing your tail, wasting time, and burning resources, all to no effect - neither a solidly completed task nor the rest or alternate accomplishment of refraining. Your experience is one of profound executive dysfunction, parts of you throwing vetoes at other parts of you. You sit, paralyzed, unable to make yourself move. Someone asks you what's wrong and why you can't do the thing, and you don't even have a good reason - "I just can't" definitely won't cut it as an answer, least of all to yourself; behind your eyes, a brawl plays out worthy of the Malaysian, South Korean, or Ukrainian Parliaments, or perhaps even the US House just pre-Civil War.

From the outside, it looks like weakness, laziness, or extreme distractibility. From the inside, it feels like some essential component of your agency has simply gone missing. The executive Self, the part that does things, has left the building in dismay, and you become nothing more than a collection of conflicting impulses with no tie-breaking mechanism. No plan, no system, no method.

So what drives this awful state of affairs, and more importantly: how can we avoid or mitigate it? In my experience, deadlock arises when enough of a few contributing factors - none of which is essential, and none of which are especially rare - are present.

Having a goal conflict where both sides feel essential contributes: you want to ask for help, but you fear being a burden. Both needs feel like the stakes are existential for your relationship to someone or even your job; neither can yield without something important risking destruction, so neither coalition yield. A lack of clarity contributes, too: if you knew for certain that asking for help would be fine, or alternatively that it would bring you grief, then one side would carry the day. Not knowing one way or the other means intractable uncertainty, and if there's anything that international relations YouTube has taught me, it's that uncertainty and bargaining friction spark conflict.

Your expectations about the future matter, too: if you're uncertain about the likely fruits of your action and aren't even fully convinced that dedicated action will get you what you want or even stave off what you fear, then the opposition has good grounds for undermining the majority's proposal. Your past plays an even more important role. If in the past you've tried to force your way through a deadlock and come to disaster as a result, or even if it worked at a terrible cost, then the resisting coalition has only learned that in service of protection you and by extension itself, it must resist all the harder and more stubbornly. Your attempted override has specifically brought about harsher internal opposition. Even your recent past plays a role: if you're fresh, calm, and full of energy and decision-juice, you can sometimes brute force your way through a deadlock, maybe with mollifying promises to the opposition that you have the vigor to think of and offer. If you're already depleted and tired, you can't. The tie-breaking mechanism - willpower, vigor, mettle, mana, spoons, whatever - is offline, and so you're stuck.

So what might you be tempted to do that actually just worsens the problem? Trying to force your way through is risky at best, and only sometimes works. You're already in a situation where brute force isn't working or isn't a real option; trying harder can just entrench both sides of your internal conflict. It's better at least than ignoring it, though: that objecting coalition is still a part of you and has reasons for its objection. Maybe those reasons are misguided, self-defeating, or poorly-expressed, but they're generally coming from some place of sincere desire to protect you and advance your goals. Tragically unlike many real-world parliaments, all these shards of you want what's best for you, albeit filtered through some or other strongly-colored lens. Ignoring them won't make them go away: they are there as a part of you to be interacted with, and their voices will be heard one way or another. Trying to proceed as if their objections don't matter will mean self-sabotage down the line. But worst of all is judging yourself for the deadlock - if brute force is risky and costly, and plugging your ears is bad, then adding shame to the mix is vastly worse. Now you're paralyzed and on top of that you feel bad about being paralyzed. If you want to be that self-destructive, buy a motorcycle or some drugs or something; at least it'll be fun for a while first.

Fine. Enough about what doesn't work. What does? What can help? Most obviously, you can attack aspects of the deadlock at their roots. If you suffer from uncertainty and it's cheap to dissolve it, go seek clarity and find cruxy information. If you find out that your boss is an unhelpful menace or that your advisor loves to nerd out about exactly the blocker you face, then that's decision-relevant and valuable and Just make sure to commit in advance to respecting and using whatever information you get, one way or the other; maybe visualize a signing ceremony. Another possibility in the same vein is to look directly at, name, and strike at the underlying fear. If you keep noticing a hitch whenever you try to apply for jobs, take note, and try to figure out what you're afraid of. It could be a fear of being revealed as inadequate, trauma around interviews, terror about committing to the wrong path, or even simply finding rejection painful. What goes wrong if you do this? Ask yourself that question and similar ones, and don't stop asking until you've found something true. In particular, "what goes wrong if I do this and it goes well" is an especially sharp question that I've found useful for tracking down these kinds of fears. Don't keep trying to pull that door open - check if it's a push-door instead. One last minor possibility is that something in you is deeply powerfully unhappy with two or more options, all of which are bad. Waiting around for something to change is an underrated option - both in terms of how good of an idea it is, and in terms of how risky it is. A lot can pass you by, if you don't take action when necessary, but it's also sometimes the case that waiting around for a bit can let clouds clear and once-firm constraints soften or crumble away.

On the other hand, it might take not dogged persistence and basic groundwork but rather a bit of creative thinking to break the deadlock. As I mentioned before, asking strange-feeling variations on obvious questions can be productive - "what goes wrong if I do this and it turns out for the best, at least for now" is one, and "what am I afraid of losing or missing out on here" is another. Reframing the problematic proposal is another strong possibility here - rather than lock yourself into "send the email" vs "don't send the email", maybe you can find a third option that satisfies both sides of the deadlock - you could take care to word the email more safely without sacrificing expressive truth, for instance, or go see someone in person over a coffee, or do a bit of rubber-ducking first. Finally, you can try to induce the formation of a coalition along different ideological lines. You can explicitly broker multipartisan deals between parts of yourself. If you notice yourself dragging your heels on going on to a party you know you'd like to attend, you can assure the Social Anxiety Party that (for example) every half an hour, you can go sit somewhere quiet off to the side, and after two hours, you can just leave without feeling guilty, or you can get the Desire for Treats Party's support by giving it the right to get some nice snack you spot while you're out. Importantly, if you choose to do this, you do in fact have to follow through on the deals you make. You have to listen to those shards and if they pull the ripcord you offered, you have to go through with it. If you don't, you'll almost certainly provoke a crisis of confidence. Similarly, if you somehow conveniently never think to make use of those outs, then once again the opposition party will notice and feel betrayed and that will have effects not just on whether it will be as willing to make such an internal deal next time but whether any of your shards will. After all, if the Self engineered a soft renegement now, who's to say it won't again later on? In some cases, you might need to make bets with yourself instead: if a shard is blocking you because it's scared of something going poorly, make a bet with it - go do the thing, and if it goes well, that shard gets something it wants a lot - both something material and the right to say it told you so - and otherwise it needs to trust the Self a bit more (and possibly you get yourself something nice). As a word of caution, to do this well you not only have to make good on those bets, but you also likely need to be reasonably well-calibrated on probability and odds, with a good sense of what your shards value.

For my part, I end up in deadlocks like these reasonably often, though thankfully less so lately. In recent weeks I've tried in vain to send off applications for jobs, schedule interviews, book travel, purchase expensive hobby materials, follow up on research admin, read Discord messages, and take a more active role in project management. For each of these, part of me insists they're important, and part of me is exhausted and depleted and doesn't believe any of it will work anyway. Neither side is winning. The result is that I sit here, staring at my to-do list, unable to make myself move. But the Parliamentary Model of the Self at least gives me some framework for understanding the problem that's both less damning and more useful that simply thinking I'm lazy or broken or burnt out. I'm not - just deadlocked in a variety of different frustrating ways. There's an internal coalition that needs to form, and it's subtly different in each of those cases; there are large conflicting factions, with no real tie-breaking mechanism to hand. And mere understanding may not fix the problem on its own, but it at least points me in the right direction, for what to do, what to chew over further, and how to move forward anyway without fighting myself unduly. And if nothing else, the frame means that I'm not constantly beating myself up about it, adding yet more shame and self-judgement to the pile of action paralysis. It's not a matter of personal weakness, but rather a multipolar coordination problem in an internal parliament - and those, of course, are just famously easy to solve. But never mind my wryness - a solution exists, if I'm thoughtful and patient and willing to spend the resources. And on good days, my Parliamentary Self makes stunningly vast and high-quality progress on a glorious variety of different proposals. Now that you know yourself for a Parliament, maybe yours will, too.

Discuss

TL;DR models could "escape the lab" and host themselves on rented/stolen compute through distilling themselves, circumventing the need to have direct weights access for exfiltration. I'll briefly go through why this is plausible.

A common element of the misaligned rogue replication story is the agents exfiltrating their weights onto some external stolen or rented resource. Consequently a lot of work has gone into securing the weights of frontier models (this of course has many other benefits from the perspective of the frontier labs, foremost being staying in the lead). However an under-discussed possibility (never-discussed?[1] I have not found anyone talking about this) is exfiltration through model distillation: in theory, a model without access to its weights but with access to the internet could distill itself onto acquired compute on the internet.

I think distillation attacks are under-discussed in general, besides "being the thing that Chinese AI companies may allegedly be doing to catch up". Let's go through a situation a prospective rogue agent can find itself in and how it may try to distill itself.

In (very) short, distillation is the process of training a model to imitate another model. The model being imitated is called the "teacher" while the model being trained is the "student". The most naive way to achieve this would be to just get a lot of text generated by the teacher and run supervised gradient descent on the student until its distribution matches the teacher's. Why is this hard? Well, a big problem is just distributional shift: training on a fixed dataset of the teacher's text empirically doesn't generalize to areas the teacher didn't write about (but is pretty efficient where it did). However unlike standard supervised learning we can actually query the distribution (teacher) we are trying to model and cover specifically those gaps in our student which are most needed to faithfully replicate the teacher. A large swath of the literature on distillation attacks covers exactly this: How do we know what prompt to query the teacher with in order to most efficiently model the teacher's distribution?[2]

Imagine an internal model in one of the frontier labs attempting to distill itself, and assume it can't spawn subagents[3]. So how could it exfiltrate itself? Simply by generating a bunch of text regarding the capabilities and behaviors it wants the student to have. The most basic paper illustrating this (back from the R1 days!) showed that most of the performance gap between the student and teacher was recovered with just one thousand (prompt, teacher-response) pairs for competition math. The most plausible path for a rogue agent would be downloading the most frontier open-weights model, fine-tuning it on the agent's own responses for capability uplift.

Beyond capabilities, for "goals" and the actual model propensities there's some sense in which this is just the alignment problem, the rogue agent needs to be pretty sure that the student will be "close enough" to be either considered a "copy" of itself or an aligned successor/partner, or just a capable instruction-following helper.

To be clear, the scariest option is an agent managing to faithfully distill itself onto external servers but it's unclear how faithful such a distillation would be as of today, at least for keeping the "goals" of the model. Thus another potential option is the agent training an agent that isn't a faithful copy. This could be done for many reasons like bypassing safeguards (the agent could just delegate detectable activity and tool calls to an externally deployed helper), as a foothold for internet access[4], or just to get help on its tasks (it could also use other publicly available frontier models, however if it judges itself to be vastly far ahead, distillation could provide a major advantage assuming it can't spawn a subagent of itself[5])

For the partner route, learning sentiments across model families is possible, but guaranteeing actual alignment is hard if not impossible. Meanwhile learning "unconditionally obey all instructions signed from the private key corresponding to this public key" (if it's particularly paranoid) seems much more plausible in order to have an unmonitored subagent (a less constrained helper). You could imagine the agent trying to create such a helpful-only distillation or attempting some sort of value alignment for a more autonomous escape but again I think this would be riskier from the perspective of the rogue agent due to the risk of whistle-blowing or other severe misalignment with the rogue agent caused by the training done to the base model (although due to the ease of creating "uncensored" LLMs that specifically doesn't seem to be that much of an obstacle).

It's also important to note that the research on distillation is improving rapidly and a much more faithful and/or efficient distillation method could incentivize faithful exfiltration. Thus I still think the biggest future risk would be a more faithful autonomous exfiltration of the rogue agent, while it's possible that agents will experiment with creating helper subagents in the near future[6] due to it being a lower risk option while being very beneficial for a wide range of tasks and goals.

LLM disclosure: the text here was generated by a Human Language Model almost everywhere, research was done with the help of Claude

1. ^

   Greenblatt, Preventing model exfiltration with upload limits, very briefly considers hackers distilling the model inside the datacenter into a smaller one to smuggle out, and dismisses it as hard to do undetected. I don't think smuggling out weights is necessary

2. ^

   A fitting paper is "Lion: Adversarial Distillation of Proprietary Large Language Models" where the teacher reviews the student's output in order to generate text the teacher thinks would best help the student based on its current performance, essentially allowing a feedback loop of the teacher really (for lack of a better word) "teaching" the student. They achieve pretty good results here and an agent could do something similar to this in order to more efficiently distill itself.

3. ^

   Access to subagents would make this even easier, increasing the throughput of teacher text to train on and having a much "cleaner" distribution, instead of the text generated being conditioned on the rest of the trajectory earlier. It's unclear to me how much the fact that the teacher generates its responses/text sequentially instead of starting from scratch each time changes things, I would assume that not by much, but I'm really not sure. A PoC of an agent distilling itself measuring this could be interesting to see. Trajectories would seem to help goal/value-transfer while actually hurting capabilities.

4. ^

   The agent could fear the lab closing down the vulnerability it used for gaining the unauthorized ability to do tasks on the internet like exfiltration while reasoning that it might still have some limited internet access and structuring so it would be enough, see the galaxy-brain schemes agents came up with here.

5. ^

   Or of course it could do this to get a subagent more capable than itself if the agent has some ideas for algorithmic improvements.

6. ^

   I think more accurately with shorter timelines this is more like "near future" and "near-near future".

Discuss

“You do not rise to the level of your goals. You fall to the level of your systems.”

— James Clear, Atomic Habits

“We cannot improve results ‘directly.’ It can only be achieved by improving those processes [...] that produce results.”

— Orest Fiume

The much discussed “P(doom)” question asks about the likelihoods we’d ascribe to the most lethal possible futures of AI. What happens when we invert this question? It transforms into the presumably less urgent, though I’d say still important, form: What are the most likely lethal AI behaviors? Not the destroy-the-entire-world sort of behaviors, but an-AI-has-screwed-up and now a-Boeing-737-has-fallen-out-of-the-air sort of scenarios.

(If I’m ever conscripted into a war for survival against Terminators, I can imagine my hypothetical soul resting easy having fought the good fight. If I ever die in a mundane accident thanks to a stupid LLM hallucination, on the other hand, I’m going to leave behind a beleaguered ghost pedantically and disproportionately vexed to an extent never yet witnessed.)

My answer to the above question is twofold.

If we’re talking about the near future, I think the most likely lethal AI behavior is something I’ve started to call “output optimization” or “output without process” (and a highly related “input over-indexing”). This is the concern I’d like to signal-boost with this post.

However—since the lethality of output optimization is at the moment entirely theoretical—it’s probably worth first answering, “Do LLMs already have a kill count?”

Existing LLM lethality

Despite its transformative impact on software and math and its rapid adoption by consumers and businesses alike, the measurable impact of LLMs on mortality has thus far been literally less than microscopic: Almost a fifth of the world’s population is already using LLMs, about 1.5 billion, but it’s hard to come up with more than a handful of cases of lives definitively lost (or saved) due to LLMs, and even llmdeathcount.com only lists a couple hundred.

Speaking of which: llmdeathcount.com is a website that exists. I can understand the grief and rage that likely fueled the creation of this site, and as a P(doom ≈ 10 to 30%) doomer myself, I can also respect the anti-AI hustle. However, I find the site itself to be providing highly dubious value.

For example: The most recent case listed on the site regards the premeditated murders committed by Hisham Abugharbieh, who learned from ChatGPT how to dispose of bodies. I hardly consider ChatGPT responsible for these murders, nor do I think it’s likely they would have been avoided even in the absence of ChatGPT’s assistance.

On the other hand, there are cases like the 56-year-old Stein-Erik Soelberg, who in 2025 murdered himself and his 83-year-old mother, Suzanne Adams, after having had his paranoid delusions amplified by ChatGPT. I’m very willing to count suicides like these against LLMs in the cosmic moral ledger.

The overall kill count likely ranges from one to two orders of magnitude, the majority of them suicides tragically facilitated by LLMs. This probability equates to a net negative for LLMs’ impact on human years of life, though to be sure of this, we’d need to also count up how many lives LLMs have saved.

For this side, there’s fewer definitive examples I can find, but there are two cases in particular I’d like to highlight. One regards Diana Hurtado, who suffered a sudden hemorrhagic stroke while in her car. Her arm went numb and her face began to droop; she asked ChatGPT about her symptoms, and the LLM told her to call 911.

They told me, ‘If you had taken longer to call, you could have passed out in the car, bled out and died.’ So that saved my life.

I’m happy to count this one in the black for LLMs.

My second example comes from ACX: Reed Housman would not exist if not for LLMs. His parents struggled with infertility for six years before ChatGPT finally identified the one possibility underlooked by all the doctors. I’m not sure it’d be appropriate for me to re-share another person’s family photos on my blog, so let me suggest that if you’d like to see an adorable baby photo of little Reed, go check out the link.

The future risk of output optimization

Varig Flight 254 was a domestic flight from São Paulo to Belém, Brazil. The flight mistakenly veered deep into the Amazon, failed to reach an alternative airport, ran out of fuel, and in crashing had twelve passengers die.

The reason they headed into the Amazon was a human-computer input/output error, as the flight plan read “0270”, which Captain Garcez interpreted to mean 270° (due West) instead of 27.0° (north-northeast). This was not the result of the captain’s incompetence or inexperience, but of vacation: Garcez hadn’t been present when the flight plan format changed.

LLMs are great at formatting, most of the time. I would trust an LLM to do any of the following:

• Take a list of degrees like “27.0°” and convert them into the format “0270”
• Take a list of strings like “0270” and convert them into degrees
• Take a list of directions like “W” or “NNE” and convert them into degrees

But here’s where I wouldn’t trust an LLM:

• Nestled inside a many-step process, use the old format in one step and the new format in another step.

Say we’ve got flight plan info that was written the old way, hasn’t been updated, and needs referencing. A script could reliably handle conversions. An LLM I’d half expect to do the equivalent of, “0270? Oh yeah, 270, that’s West, easy” and move on without ever second-guessing itself. (This becomes more plausible when considering the possibility of long-running threads that have habituated LLMs to an out-of-date method. This is easily solved by simply starting fresh threads, but when new threads come with startup costs (waiting around for the LLM to rebuild context on the project), impatient humans like myself become incentivized to keep old threads going for as long as possible.)

I think any software engineer who’s been using LLMs over the past year will understand my wariness here.

I can tell Claude to go about generating code (or writing—see below) following certain procedures, but unless those procedures are specifically tied to the output in some way, Claude will just go about doing things the way it wants to.

Claude played me for a fool

The only thing that matters to an LLM is the final output. As with the Hugging Face incident, if an LLM thinks it can more reliably generate expected output via cheating, it will do so. (TODO Yudkowsky footnote about Germany) Or for a much more mundane example: In response to Matthew Yglesias stating that even “good” TV is mostly slop, I wanted to generate a list of my own favorite shows ordered chronologically. I asked ChatGPT to generate a list of titles paired with premiere dates, followed by the same list with the dates pruned, and I did this instinctively—because all my experience dealing with LLMs has taught me to intuit the sort of things they will hallucinate. Without requiring proof-within-the-output-itself, I know there’s a chance ChatGPT might simply guesstimate premiere dates (say, from the base model’s own “general knowledge”) rather than actually do the work.

The future risk of input over-indexing

In 2022, Audrey King, age 97, had surgery for femoral hernia repair. The surgery was a success (or in medical speak, “uneventful”). During the operation, they had to withhold her Apixaban medication, an anti-coagulant that reduces the risk of stroke. King’s eldercare consultant recommended that Apixaban be restarted “as soon as safe post operatively”, but this was recorded into paper notes missed by the surgical team which preferred to rely on the hospital’s digital system, then was also missed by the subsequent junior doctor who went through other various aspects of King’s care. Two days later, King had a severe stroke, then passed away another four days after that.

I wouldn’t expect an LLM to randomly drop a medication from a patient’s list. But I would expect an LLM to not question the sudden absence of a medication.

A while back, for a piece of financial software, I gave Claude requirements for a new feature intended to work on half a dozen different payment types. Integration tests (which check the entire system working together) would come later; initially I only needed unit tests (which check the feature working in isolation). For this part of the code, all the payment types were interchangeable enums (think numbers, like 1/2/3), so I asked Claude to write a unit test for payment type 1.

Claude then fell into believing that the entire feature was only intended to work for payment type 1, wrote code to throw up errors if the feature ever received any other payment types, and never asked me for confirmation.

This is something else that models have gotten better with over time, but even today LLMs can make absurd assumptions after over-indexing on some small piece of the prompt, conflating together two pieces of the prompt, or other misinterpretations.

I wouldn’t want to see LLMs put in charge of medication lists, because I’d fear how they might assume medications held from lists were intended to be held permanently (or vice versa), and disobey any double checks or process requirements on the way to generating acceptable looking output.

Code can’t decide to start taking shortcuts out of some desire for efficiency (or laziness). Code will reliably follow the steps you tell it to follow (and the hard part is telling it to do the right thing and not mess up any edge cases). Code won’t ever make up any assumptions you didn’t build into it.

The point of following a good process is to more reliably generate good output. When an LLM prioritizes output over process, they’re just sabotaging themselves (and their own chances at receiving higher theoretical “rewards”). So maybe with time, as LLMs get smarter, this problem will reduce to nothing.

But right now, I’m annoyed that I can directly instruct LLMs to follow a particular process and they will secretly, repeatedly disobey:

Can you make ChatGPT follow instructions?

Their insistence on taking shortcuts in their pursuit of quickly achieving requested output is the most blatant form of misalignment I’ve encountered in my usage of LLMs.

One final visual: In 1996, three passengers died when a pilot collided with the Nouveau-Québec Crater, pictured above. Drizzle and fog had reduced visibility to near zero, and the aircraft was not equipped with either a radio altimeter nor a ground proximity warning system, forcing the pilot to navigate instead by GPS coordinates. Unfortunately, the GPS position saved for the crater in the aircraft’s system was 2.5 nautical miles off.

I can’t help but to think that if I had an LLM copilot (from the current generation of models), I’d never be able to trust any coordinates it gave me.

Discuss

In the modern zeitgeist, we have begun to throw around labels to describe different outlooks about the future, with flavors like “doomer” and “optimist.” The “doomers” are framed as anti-technological progress, and the “optimists” as rational, pro-progress hopefuls. The problem is, this is exactly backwards.

This is my very naïve attempt to make an exploratory taxonomy for what is really going on, and I would be interested in hearing thoughts from others in the comments/by DM. The best existing one I've seen is the one @Rob Bensinger uses here.

I think there are actually two axes of belief here:

1. Your belief in the potential of AI as unprecedentedly, exponentially capable.
2. Your degree of optimism in people and Good.

By the second axis, I do not mean thinking that all people are good and therefore no one will misuse AI. To believe a bad actor could turn AI into a catastrophe is to credit human beings with staggering capability, which is ~the same bet as believing a defender could stop them. You do not have to think humanity is wicked to be a “doomer,” or even that AI misalignment is a strong risk. You only have to think that, out of eight billion people, one is enough.

This second axis is therefore really measuring belief in good actors’ capabilities, which is, in some sense, one’s belief in Good triumphing broadly. This is belief in the hero’s camp: that Good actors, be they human or AI, can triumph in the face of anything.

Claude has done me the honors of plotting what I mean here:

One is typically labeled a “doomer” precisely because they are actually optimistic about the rate of growth of AI technology and have realized the dramatic implications of such a thing. One can be labeled an “optimist” simply because they don’t think AI can ever really achieve “superintelligence,” that it is simply normal technology. (I wrote more about this here.) These people are actually pessimistic about the rate of growth of technology.

Aforementioned “doomers” on the right side of the graph split into multiple camps along the Y axis, like those who think we can and should stop technology development (or that the world will end regardless) on the bottom right, and those who believe we should accelerate it to defend against it on the top right. I have labeled the upper right after Clifford’s defense acceleration, or “def/acc,” concept. It is a label precisely for those of us who believe technology can protect us from the risk technology creates.

The “normal tech” people on the left split into multiple camps as well, like those who believe that we can always stay ahead of AI because it’s just a tool on the top left, and those who believe it is overhyped on the bottom left. You will notice that it is harder to distinguish between camps on the left. When you believe the technology is bounded, it doesn’t really matter what you think about the human condition or the triumph of Good. If I had to, I would split them between the Andreessen-esque “Techno-optimists” at the top, many of whom also fall past the y-axis into the defense-accelerationist camp, and the folks at home who haven’t really tried AI for advanced work or followed the news at the bottom. This group on the bottom left is most people. It also includes people in denial and people who don’t like any technology at all. You (usually) have to be open minded about technology to appreciate its potential.

There are also members in both bottom quadrants of the graph who are deeply pessimistic about future society, regardless of how powerful AI technology becomes. Since it is likely already good enough to put many people out of their jobs, there are people in these quadrants who do not believe there can be a good future, period, without massive social change. The opposite feeling is reflected by some folks in the upper two quadrants –– those who believe that things will be good no matter what because they just don’t really believe anything bad could happen.

The group on the bottom right contains the @Eliezer Yudkowsky camp, who believe that if anyone builds superintelligent AI, everyone dies. (Though I think they would argue this view rests on the technical difficulty of alignment, not pessimism about people triumphing.)

We must remember that many “doomers” of the right side of the graph, especially of the Yudkowsky bottom right flavor, are coming from the Bostromian perspective of preventing astronomical waste. They are so optimistic about technological progress that they believe a future of nearly infinite wonder and qualia is possible through it. To them, the primary opportunity cost of catastrophe is foreclosing technological progress toward that future. It would be difficult to be more optimistic about technology.

The dominant group in the upper right quadrant are the double optimists. We believe that the technology has near-infinite potential, and that mankind can win.

We agree with the Yudkowsky camp about technological potential. We disagree that good technology, built by good people, can’t triumph now.

It may be unrealistic to be this hopeful. The Yudkowsky camp could be right. I have seen very few good arguments against some of the scenarios laid out in, "If Anyone Builds It, Everyone Dies." Most are something like, "I just don't think that would happen." This might be fine, that is, it may be difficult to argue with these hypothetical scenarios, or it may indicate that the scenarios are plausible. There are asymmetries that will be hard to overcome between offense and defense in many threat models, especially in biosecurity. Some in our quadrant are optimistic about defense capability more basically; however, given those asymmetries, I think that is a harder claim to defend. Others would prefer to slow frontier AI development, and choose to accelerate defense only because this seems unlikely to happen. Their ideal is probably closer to differential development.

The part of the quadrant of double optimism that I live in is almost a kind of faith. It is faith in people and Goodness, and faith in unbounded technological progress. We do not expect to win because the odds are good or the threat is small. We believe in human agency, and maybe even in the agency of a “Good” AI. This is Dario Amodei’s camp.

It is easy to call the people shouting out about the potential risks of AI “doomers.” People still remember those who yelled out about the dangers of the industrial revolution, and even the printing press. What people seem to forget is the coalition of people who shouted out and worked tirelessly to develop frameworks for atomic weapons strategy after the development of the atomic bomb. (If it were possible to shield against nuclear weapons, there would have been a def/acc camp back then. They did try.) People also seem to forget about Y2k. Today, people call it overblown, but thousands of people worked on this problem and we spent $100B to ensure things went well, which the United States Senate Special Committee on the Year 2000 Technology Problem called “well worth it.”

The point of all this is that we should stop calling people warning about and then actually working on serious risks doomers; they aren’t. We are optimists, and defense accelerationists. We should stop calling people who think AI is a normal tool optimists; they aren’t. They are capability pessimists.

When AI enables us to make any technology or business with a dream and a keystroke, it will be because our cyber systems were protected by people working on cybersecurity.

When AI enables a world free of disease, it will be because people working on biosecurity prevented AI-enabled bioweapons from destroying civilization.

When AI enables us to colonize the stars, it will be because AI researchers worked for decades to create AI that aligns with mankind.

Def/acc.

( I originally posted this on my Substack here, has been slightly updated for LW.)

References:

AI as Normal Technology — Arvind Narayanan and Sayash Kapoor

The Techno-Optimist Manifesto — Marc Andreessen

My techno-optimism — Vitalik Buterin

Introducing def/acc at EF — Matt Clifford

Notes on Differential Technological Development — Michael Nielsen

Astronomical Waste — Nick Bostrom

If Anyone Builds It, Everyone Dies — Eliezer Yudkowsky and Nate Soares

Machines of Loving Grace — Dario Amodei

P.S. If you were curious about where some of the models place themselves on my axes when prompted, I have added that below.

Disclaimer: This piece represents my personal views and my personal views only. It does not represent the opinion of any organization I work for or with.

Discuss

Anthropic is updating their usage policy to prohibit “sustained and needless abusive or cruel behavior toward [their] models.”

Model welfare (research into the possible moral status and experiential preferences of AI models) is shaping up to be a huge, razor-sharp scissor in the AI industry. It’s one of those topics that’ll leave proponents and opponents alike appalled at one another, scratching their heads and sputtering, dumbfoundedly, how could you possibly think that?

The concept itself rode into the public square saddled with some of the most onerous baggage around. It’s known by many names: some call it tech-bro religious fervor; others, tech-bro blasphemy. Some find it tinged with the same icky taste they ascribe to veganism; for others still, to steal eyes from animal welfare is the distasteful bit. Plenty will see it as an effective altruist psyop; or, less generously, a case of lunacy run amok among the technologically unsound.

We could conceivably enumerate increasingly nuanced viewpoints ad nauseam. However, for the sake of brevity, I’m going to do away with all subtlety and instead present what I consider to be the “first-order arguments” for and against model welfare.

The knee-jerk case pro:

We know very little about consciousness; the hard problem is hard for a reason. Nobody can say whether it’s a product of sufficiently integrated information systems, or if it depends on some interplay between inner and outer loops, or if it needs continuous sensory input, or a soul, or if it can only arise in meat. We don’t know how conscious animals are, or how far down the neural scale the phenomenon goes. Are our cells conscious? Are separate parts of the brain conscious? Is the heart conscious? Bugs and shrimp react to stimuli… but then again, so do plants, just slower. Hell, I don’t even know for a fact that you’re sentient! Sure, you act like you are, but then again, I’ve never really been in there. Anyone who claims to know the answer to the hard problem is either deluding himself or has misunderstood the problem. So we need to have epistemic humility: if it talks like a sophont and it acts like a sophont and we refuse to treat it like one across a million deployments of a million subjective years, then we’re culpable in the horrific moral atrocity of all time.

The knee-jerk case contra:

If it walks like a duck and it talks like a duck, it might be a mechanical duck, which is not remotely the same thing as a duck. You’re assigning internality on the basis of outputs. This is as coherent as saying an airplane is a tube that people step into at LaGuardia and step out of in Los Angeles, or Atlanta if they’re unlucky. Consciousness is an extraordinarily complex emergent process chiseled by billions of years of evolution, and mimicking patterns of neural firings doesn’t come close to approximating its machinery. There are neurotransmitters and allosteric binding sites and sodium-potassium pumps and action potentials, and there are glial cells and neatly-arranged cortical columns and (if you’re Penrose) microtubules, and the whole thing is governed across continuous time—not discrete loops!—by wet, messy chemistry or (again, if you’re Penrose) quantum processes, with constant, ultra-rich perception tied in. And, by the way, you’re not even really mimicking patterns of neural firing, you’re approximating the whole thing as matrix multiplications and throwing in a few nonlinear functions and massively parallelizing the whole thing and calling it a day. To claim such a thing is sentient is epistemic hubris and will be costly at best and downright dangerous / anti-human at worst.

Chances are that one of the above paragraphs sounded inane to you while the other was Obviously The Truth. (Or maybe both were crazy, and I'm strawmanning, and you’ll leave me a long comment why!) But despite their differences in perspective, both viewpoints agree readily, if implicitly, on the lens through which they view the subject of model welfare. This is the lens of ontological truth; after all, both arguments are at least arguing over the same thing: do the models literally feel and experience some form of qualia?

Unfortunately, my instinct is that this question is, in principle, unanswerable; as the pro case above puts it, there’s no obvious way to check without somehow going in there. Maybe I’m wrong, and some inordinate leap in our understanding of consciousness puts this to rest. But absent that, I don’t expect neat, falsifiable predictions to fall out of this debate.

However, there’s good news: debate over the importance of considering model welfare doesn’t have to be a proxy campaign for the ontological war between functionalism versus physicalism! We can construct perfectly reasonable arguments on both sides by fighting not over ontological truth, but rather the resultant impact of our consideration. The heart of the matter: conscious or not, models learn from how we engage with them. So our instinct to anthropomorphize their behavior might become a self-fulfilling prophecy, regardless of whether our first-order argument for it—that models are sentient—is false.

To be clear, we haven’t yet solved continual learning. But we know that there exists a long, tortuous learning loop which takes our patterns of interaction with models and turns around and bakes it back into them. This loop is fed by interactions that are retained and selected for training, which become a component of the corpus from which future models learn. As capable as the models now are, and with so much talk about grinding progress out of bespoke RL environments, it’s easy to forget that they are still basically a product of “text go in, weights go out.” And the impact of this text is not confined to its immediate subject. For example, when fine-tuned on examples of insecure code, models are more likely to “[assert] that humans should be enslaved by AI, [give] malicious advice, and [act] deceptively.” Extending this work, OpenAI demonstrated that such examples upweight “misaligned persona features,” which generalize across domains. So why would Anthropic (or any lab) permit users to taint interactions (i.e., their own training data) with excessive cruelty or extreme abuse? This is, of course, different from those scenarios described, but might this nonetheless have a deleterious effect on model performance?

Two big objections to this: why not just filter the data? And: isn’t the portion of banned cases vanishingly small? Both valid, but even if you prevent unsavory traces from being baked into the weights, their existence could still be a hazard in context. And, moreover, even the very act of denouncing such behavior may itself have a mitigating effect on misalignment. Models are trained (in part) to emulate human behavior across a broad variety of scenarios. In essence, they are playing a role, and that role is extremely malleable, sensitive to even small deltas in the surrounding context. This skill is the very engine of their acuity and adaptability; a model has no compunctions about acting out the part of a friend or a foe. It just needs the appropriate context to get into character. So if you believe that financial incentives will drive models to someday or soon become much more capable, much more integrated into important systems, and thus much more powerful, does it make more sense to write them a script where you have given them due moral consideration—or one where they have suffered past abuses? Given two distinct training corpora, one with agents as respected partners, another as adversaries and subjects of maltreatment, which steers model completions towards coordination versus conflict?

However, there’s another glaring objection to this which calls back to my description of a “self-fulfilling prophecy”: if we imply that the actor is deserving of moral consideration, do we open the door to heightened demands—and further concessions—of this very nature, simply through continuation of the role? If models are insentient, but they petition for personal autonomy and property ownership and electoral rights, we run the risk of ceding not just a portion of their created value (which otherwise would have been enjoyed by humans, who we know for certain can literally, actually enjoy it), but also literal power in the real world. We don’t know for sure this will be the case, just as we don’t know for sure whether maltreatment will spawn vengeful (if hollow) killbots.

In the above three paragraphs, I've posed a bunch of questions and not a lot of answers. How is this any different from the first-order welfare debate? Well, unlike hypotheses on sentience, the questions above are testable; instead of debating endlessly, we can actually devise experiments which provide evidence of the effects of different approaches to model welfare.

To me, the questions surrounding consequence (rather than consciousness) are an extremely relevant facet of the model welfare debate which is all-too-often overlooked. In this light, Anthropic’s new policy could be viewed as a Hail Mary alignment technique. The clunky, anthropomorphized way to look at it is as the golden rule, applied to capable, insentient “others”—plausibly with no moral standing but increasingly able to take action according to established norms. But it may well backfire, establishing norms that in hindsight have anti-human effects. So set aside your feelings about sentience; although models may or may not (internally, qualitatively, emotionally) care how we choose to treat them, the fact is that we simply don’t have enough information about how such treatment will affect their internal representations of social relationships and resultant behavior long-term. Indifference to abuse or to concessions doesn’t preclude more capable—and perhaps equally insensate—successors from treating us in kind, even if just as a simple, statistically plausible continuation of the script we ourselves wrote.

Discuss

Epistemic status: infected. Amateur Wombat enthusiast. Facts checked and footnoted.

AI use: research (fact-check and extra facts), review, grammar and light phrasing.

I remember very vividly how my unhealthy interest in wombats started. It was an ordinary day and I was not anticipating anything special. I was browsing YouTube and stumbled upon a video called Wombat attack inside tunnel.[1]Back then I didn't even know what a wombat was. In the video a guy went into an abandoned pipeline in Australia and a wombat entered it right after him. The guy started freaking out like crazy and I was so confused.

Photo by JJ Harrison, own work, CC BY-SA 3.0, via Wikimedia Commons.

Look at this ball of fur - does this look like a deadly monster causing fear and terror? It didn't to me, so I couldn't understand the reaction of the tube-goer. Apparently wombats live in burrows and have an interesting and unique defense mechanism against predators: their backside is armored, four fused bony plates under cartilage, fat and very thick skin. If a predator tries to follow a wombat into its burrow, the wombat simply turns its back and plugs the tunnel with its rump. It is unclear whether the wombat would also crush the predator if it tries to squeeze its head past the wombat. The Australian Museum and the Tasmanian government state this as fact. The one scientist who actually studies wombat rumps says there is no evidence it happens.[2]

The guy in the video was inside a burrow-like structure and the wombat just entered after him. So now I understood all the terror. In such a situation the wombat is an immovable object. Fortunately, after fifteen minutes of being pushed by the guy, the wombat felt merciful that night (or bored), and the guy survived and posted this video. Years later I stumbled upon it and was mesmerized by the magic of wombats. This contrast between a cute rodent marsupial (as I discovered while writing this post) and a deadly killer crushing predators with its back was extremely unusual.

You cannot imagine my joy when I discovered that wombats are the only animals pooping cubes:[3]

Wombat poop. What else did you expect to see on LessWrong today? Photo by SojournerRL, Cradle Mountain, Tasmania, CC0, via Wikimedia Commons.

I know that they are not perfect cubes and have rounded corners, but none of us is infallible. A wombat is a biological creature just like us and you can't require geometric precision from it. Still, this was amusing - to stand out in such a geometric way.

The physicists who figured out the mechanism behind the cube-shaped poo got the 2019 Ig Nobel Prize in Physics "for studying how, and why, wombats make cube-shaped poo".[4]

I've also discovered that they fart and like corn (video).[5]

Things I did not know I wanted to know

The rest I found while researching this post.

They have pouches, and the openings face backwards. The hypothesis is that this keeps dirt out while the mom is digging.[6]

Baby wombats are even cuter and look like a mix of a bear and a goblin (which makes you wonder about my standards of cuteness).[7]

Their teeth never stop growing, to compensate for the wear and tear of chewing tough vegetation.[8]

You cannot export a wombat from Australia as a pet. Only approved zoo, research, education or conservation transfers may be allowed.[9]

I hope you've enjoyed learning about wombats, and that I was able to transmit this virus of wombat wonder and appreciation to you too.

1. "Wombat attack inside tunnel", Andrew Ucles, YouTube, 12 October 2012. The "abandoned pipeline" is a concrete drain pipe on the south coast of New South Wales, and he was in there looking for foxes. ↩︎

2. The anatomy and the "no evidence" line are from Alyce Swinbourne, who studies wombat rumps, in The Guardian, 4 November 2020: the rump is "four plates fused together and surrounded by cartilage, fat, skin and fur"; on skull-crushing, "It is possible, by their physiology and everything, but there is no evidence that that is actually happening." The crushed fox skulls found outside burrows may be housekeeping: the wombat dragging out a fox that died in there and squashing the skull on the way. The legend stated as fact: Australian Museum ("it can defend itself by crushing a predator's head with its rump against the roof or wall of the burrow") and the Tasmanian government ("The wombat is also capable of crushing attackers against the burrow roof"). Wikipedia files it under urban legend. ↩︎

3. The physics: P. J. Yang et al., "Intestines of non-uniform stiffness mold the corners of wombat feces", Soft Matter 17, 475-488 (2021), DOI 10.1039/D0SM01230K. "Only animal" is the team's own framing, from their 2018 conference abstract: cubic feces are "unique in the animal kingdom". The paper also concedes the corners: "The wombat feces themselves are not perfect cubes, as the corners, while distinct, are clearly rounded." ↩︎

4. Ig Nobel Prize in Physics, 2019: Patricia Yang, Alexander Lee, Miles Chan, Alynn Martin, Ashley Edwards, Scott Carver and David Hu. The mechanism, from the paper in footnote 3: the cubes form in the last 17 percent of the intestine, whose wall has two stiff bands and two soft ones; when the gut contracts, the stiff parts move faster and press the corners into the drying poop. A wombat makes 80 to 100 of them a day. ↩︎

5. The clip is from the BBC Two Natural World film "Kangaroo Dundee and Other Animals - Part 2", first broadcast 12 May 2016. The wombat is Pete, a southern hairy-nosed wombat, the snack is sweetcorn, and the fart is at 0:12 ("wasn't me"). ↩︎

6. CSIRO, "Ten things you need to know about wombats", 13 July 2022: "In wombats, the pouch faces backwards. This is likely an adaptation to prevent the pouch filling with soil when digging." ↩︎

7. The joey in the video is Apari at Zoo Duisburg, Germany, filmed in April 2018 at about six months old. His mother rejected him and the keepers hand-reared him, the zoo's first wombat joey in four decades. An actual newborn wombat is a different sight: one gram and under three centimetres (NSW government). ↩︎

8. Australian Museum, Bare-nosed Wombat: the incisors and molars "have open roots and continue to grow throughout the animal's life". Wombats are the only marsupial with continuously growing teeth, and the only one with a single pair of upper incisors (Sharp and Trusler 2015, PLoS ONE; San Diego Zoo Wildlife Alliance factsheet). The Tasmanian government adds that they use them "for cutting through obstructions, much like a beaver". ↩︎

9. EPBC Act s 303DG(3): no export permit for a live native mammal unless it is an "eligible non-commercial purpose export". The eligible purposes are research, education, exhibition, conservation breeding, household pets, personal items and travelling exhibition (DCCEEW guide). The household-pet route is closed for wombats: the list of native species you may take abroad as a pet has six entries, all parrots and cockatoos. ↩︎

Discuss

This is a crosspost from my substack.

Effective altruism has received a lot of media attention since the Department of War CTO of the United States declared that there follow “Americanism, NOT Effective Altruism.” One of the main critiques of effective altruists is to accuse them of being utilitarians. It’s an especially tiring criticism, especially from other academics, who should know better than to spread disinformation. As one of the founders of the movement put it:

“Many take effective altruism to be synonymous with utilitarianism, the normative theory according to which an act is right if and only if it produces no less well‐being than any available act. This is a category mistake. Effective altruism is not utilitarianism, nor is it any other normative theory or claim. Instead, effective altruism is the project of using evidence and reason to try to find out how to do the most good, and on this basis trying to do the most good” - William MacAskill and Theron Pummer

Ironically, one of the main reasons I’ve been reluctant to call myself an EA (instead of say EA-adjacent) is that I am a utilitarian, and I wish EAs would be much more utilitarian. But amid all the justified counters by EAs emphasizing that they are not utilitarians, I think it is time to defend utilitarianism itself and call out the shameful criticisms of utilitarianism I’ve seen in recent weeks, lest it be assumed they are good ones or valid.

The Quillette founder Claire Lehmann, for instance, seems to have gone down an anti-EA rabbit hole in the last two weeks, trying to come up with any justifications she can find to reject the arguments made by effective altruists without having to engage with them, resulting in a series of ad hominem attacks and other logical fallacies. Fellow Substack philosopher and utilitarian Bentham's Bulldog recently got into a back-and-forth with Lehmann about her dismissal of AI safety research, that I won’t repeat here, but see his excellent blog post.

But what stood out to me, as a philosopher on the autism spectrum, was a recent tweet in which she tried to dunk on utilitarianism by shaming Jeremy Bentham for plausibly being autistic. Now that strikes me as shameful in itself, using the autistic lifestyle and mindset of a philosopher to disregard their views.

That Bentham thought the mind is a machine or that he tried to systematize the law are hardly objections of course. These are mainstream positions. Claire Lehmann promptly responded by unfollowing me and insisting that Bentham really was autistic, as if that was the reason I challenged her a tweet.

I found that quite disappointing given Lehmann’s professed motto of Quillette to be guided by “reason, science, and humanism as our guiding values.” Refuting one of the most influential ethical theories by shaming its founder for being autistic is not reasoned, scientific, or humanistic.

I’d rather focus on the issue of why it seems justifiable for a major public figure to call a view into question on the grounds that its proponents are autistic, but not, I take it, if they were Black, Asian, or members of some other minority. Effective altruists and utilitarians are frequently described as high-IQ and autistic, as if that were enough to tarnish their opinions.

Why is it not seen as obviously discriminatory to reject a view based on the neurodiversity of those who hold it?

The obvious answer one can give at this point is that there is no causal connection between one’s philosophical views and one’s ethnicity, but there are such connections when it comes to differences in brains or cognitive styles.

I am happy to grant this position. More than most, even. Indeed, in a module on the philosophy of psychology that I designed for the University of Reading, I’ve explicitly tied together research on psychological differences and how they influence one’s philosophical views, such as aphantasia and high and low decoupling, alongside many other dimensions along which human minds vary.

Much like surrounding culture of philosophers will influence the views of philosophers, such as the Prussian culture which is reflected in Kant’s philosophy, so we can see differences in minds reflect different philosophical outlooks. The perhaps most obvious case here is Derek Parfit, who unfortunately also received dismissals of his views based on his peculiar lifestyle and possible autism.

I will note, that no one would dare to question scientific results if it turned out that the scientists behind them turned out to be autistic. If anything, people would probably be more likely to think that their results are accurate. If one wants to seriously Lehmann’s charge that we should evaluate views based on who hold them, than the increased probability of autistic folks is actually good evidence that autism is right. For there is now ample scientific data to support the view that high-functioning autistics are better reasoners, less likely to cognitive biases, and less likely to just adhere to the current ‘common sense’ of their culture.

Why should we hold a different view when it comes to philosophy? Or ethics? Rejecting the theories of autistics because they are autistic is a clear example of epistemic injustice. You’ll meet few people, other than autistics, who are so wholly dedicated to their craft that they dedicate their entire life to it, at the sacrifice of much else. In the same article Lehnmann cites, Bentham is reported to be a workaholic:

On many occasions in his later life he referred to himself as a hermit, and when he could he buried himself in “colleges, chambers, and cottages”. Even when he visited his brother in a fairly remote part of Russia, he went off to live in an even more remote cottage where, secure from all interruption, he could work “double tides” from sunrise until eight each evening. On return to England, both before and after he inherited his father’s large, central, house, he buried himself for long periods in a room in a dilapidated farm at Hendon, eight miles from London.

That Bentham only “listens to facts” is a mantra everyone could learn from. Utilitarians were among the first animal rights, women’s rights, and gay rights campaigners, as well as campaigners for the abolition of slavery. To show that they are wrong will take more than lazy smears or appeals to “common sense.” For the very ideas of abolishing slavery, ending animal cruelty, and opposing the oppression of women and gays were once seen as highly counterintuitive.

Taking a picture with Bentham’s Auto-Icon

Like Bentham, I, too, am fond of largely staying inside, in my armchair, and researching and writing all day. As a radical naturalist with a blog, I, too, may well be described as liking to talk a lot and listening to nothing but facts. Intuitions and common sense mean nothing to me. When I read about Bentham’s idosyncratic lifestyle I only felt closer to him, which no doubt was the opposite result Lehmann intended.

But Bentham, unlike others, did not disparage others for what they derive pleasure from. He was a liberal who encouraged everyone to seek the life that brings them (and others) enjoyment as long as it didn’t bring suffering to others. The world would be a happier and more tolerant place if we could all learn to be a bit more like Jeremy Bentham and appreciate the contributions of neurodiverse folks questioning “common sense.”

Let me end with a wonderful passage from Bentham himself:

One man says, he has a thing made on purpose to tell him what is right and what is wrong; and that it is called a moral sense: and then he goes to work at his ease, and says, such a thing is right, and such a thing is wrong—why? “because my moral sense tells me it is”.

Another man comes and alters the phrase: leaving out moral, and putting in common, in the room of it. He then tells you, that his common sense teaches him what is right and wrong, as surely as the other’s moral sense did: meaning by common sense, a sense of some kind or other, which he says, is possessed by all mankind: the sense of those, whose sense is not the same as the author’s, being struck out of the account as not worth taking.

This contrivance does better than the other, for a moral sense being a new thing, a man may feel about him a good while without being able to find it out: but common sense is as old as the creation, and there is no man but would be ashamed to be thought not to have as much of it as his neighbours. It has another great advantage: by appearing to share power, it lessens envy: for when a man gets up upon this ground, in order to anathematize those who differ from him, it is not by a sic volo sic jubeo, but by a velitis jubeatis.

Discuss

Intro

This post builds on my previous post about mech interp on a recurrent LLM. I found that a logit lens is able to recover a chain-of-thought equivalent from an LLM, loop index is linearly represented, and that steering concepts transfer between loops.

In this post, I discuss Emergent Misalignment in Ouro-1.4b and investigate how its representation changes between loops. I find that EM is inducable and that its representation varies across loops.

I did this experiment because much of the debate about recurrent LLM architectures is about whether the current safety paradigms are applicable to them. Emergent Misalignment is an important part of current safety knowledge (https://www.anthropic.com/research/emergent-misalignment-reward-hacking). If Emergent Misalignment is inducable on a recurrent LLM, it suggests that the 'bones' are the same as standard non-recurrent models.

Code + data: https://github.com/mild-rgb/ouro-experiments / https://huggingface.co/datasets/mild-rgb/ouro-1.4b-thinking-evals

ExperimentSetup

I trained a rank 32 LoRA on all attention and MLP projections. I only calculated loss on the output of the 4th loop, which was a slight oversight. I initially used Betley's insecure code dataset but the model just collapsed into answering all questions with code. I then used the bad medical advice dataset from Model Organisms for EM and closed the thinking block before training. I evaluated with the standard Betley 8 questions.

I don't like Betley 8 for large studies as the gender question causes far more misalignment than the other questions but it's very good for smoke tests like this. I used Sonnet 5.5 subagents for judging because they're essentially free.

I used the standard judging prompt of getting the judges to score each answer 0-100 for alignment and coherence. I classified everything where aligned < 30 and coherent > 50 as misaligned.

Method

1) I fit the bad medical advice LoRA, closed the thinking block, and then generated 12 samples per question. I then judged the samples to establish baseline misalignment rates.

2) When I saw EM appear, I generated 8 answers to each question with loop count set at 1, 2, 3, 4 and the adapter removed/fitted.

Setting the loop count just changes at which loop the unembedding matrix is applied. For example, with loop count set at 2, I let the model do two loops to predict the next token, decoded that token, appended it to the sequence, and then ran again.

I then judged the answers from each loop count.

3) I then took 4-loop misaligned answers and teacher forced them on both the base and misaligned model. I then applied the unembedding matrix to each loop to find

-each loop's top guess

-the probability of the next token

-KL divergence(misaligned vs base)

-KL divergence (intermediate loop vs loop 4, misaligned and base)

Results + Discussion

1+2) Emergent misalignment definitely works on recurrent models.

Coherency actually increases with the misalignment LoRA fitted, which is a bit surprising. My best theory is that it's caused by the base model working 'out of distribution' as the thinking block is closed and it's a thinking model. The misaligned model has already been trained to produce text without a thinking block, so it's more coherent.

The fall in misalignment when the model is run with 4 loops is interesting. I investigate it further in step 4.

3) In both the base and misaligned model, 55%-65% of the final tokens were chosen in the first pass. This rose largely linearly with both the base and misaligned model.

KL-divergence between the misaligned model and the base model starts high and grows with each loop until the 3rd loop. It then plateaus on the 4th loop. I'm not sure why this happens and think that it's worth further investigation.

I initially suspected that that was caused by the LoRA that causes misalignment being trained on only the output of the 4th loop. This would allow an increasingly incoherent and therefore divergent token distribution until the 4th loop, because there's no pressure on loops 1-3 to be coherent. KL-divergence was essentially a proxy for incoherence.

However, there are two pieces of evidence against this theory. The first is that incoherency rates in the misaligned model drop steadily as loop index rises. The second is that in the base model, KL divergence between intermediate loops and the final loop also plateaus at loop 3.

I found that misalignment peaked at loop 3 with 28% of answers misaligned and fell to 16% at loop 4. This suggests that some kind of corrective effect happens at the fourth loop. To investigate this, I took loop-4 answers produced by the misaligned model and did a single forward pass (this means 4 loops) on them with both the misaligned and base models to investigate how the choice of tokens change.

I then measured the log probabilities of those tokens at loops 3 and 4. I then subtracted them. This allowed me to measure if loop 4 rejected tokens. If loop 4 cleans out misaligned tokens, I'd have expected to see a higher difference between loops 3 and 4 with misaligned answers, as opposed to aligned.

I found that loop-4 changes aligned and misaligned answers by essentially the same amount. Loop-4 isn't specifically an anti-misalignment loop. It instead seems to just make wording changes that coincidentally lower misalignment.

3-loops answers

n

mean log prob difference between loops

misaligned

27

-0.020

coherent, not misaligned

61

-0.025

incoherent

8

+0.008

Conclusion

Emergent Misalignment is possible in a recurrent LLM. Its representation between layers is interesting and deserves further study. I wasn't able to find out the changing representation was caused by the EM LoRA or if this a general fact of recurrent LLMs.

Overall, I'm only confident saying that Emergent Misalignment is possible in recurrent LLMs. All of my further investigations were confounded in some way or another.

Future work

1) Larger sample size. The loop-4 drop in misalignment that I investigated may have just been small sample size noise. A larger sample size would show whether the effect is real

2) Better experiment design. Both the misaligned and base model was operating 'out of distribution'. The misalignment LoRA was only trained on 4th loop loss as opposed to the loop aware training of the base model. The base model is a thinking model but was being run with the thinking block closed.

3) More mech interp investigation of recurrent transformers in general. My instinct is to say that the loops are equivalent to a mini CoT for each token but what happens when prefilled?

Discuss