Новости LessWrong.com

The Big Nonprofits Post 2025

27 ноября, 2025 - 19:20

Published on November 27, 2025 4:20 PM GMT

There remain lots of great charitable giving opportunities out there.

I have now had three opportunities to be a recommender for the Survival and Flourishing Fund (SFF). I wrote in detail about my first experience back in 2021, where I struggled to find worthy applications.

The second time around in 2024, there was an abundance of worthy causes. In 2025 there were even more high quality applications, many of which were growing beyond our ability to support them.

Thus this is the second edition of The Big Nonprofits Post, primarily aimed at sharing my findings on various organizations I believe are doing good work, to help you find places to consider donating in the cause areas and intervention methods that you think are most effective, and to offer my general perspective on how I think about choosing where to give.

This post combines my findings from the 2024 and 2025 rounds of SFF, and also includes some organizations that did not apply to either round, so inclusion does not mean that they necessarily applied at all.

This post is already very long, so the bar is higher for inclusion this year than it was last year, especially for new additions.

If you think they are better places to give and better causes to back, act accordingly, especially if they’re illegible or obscure. You don’t need my approval.

The Big Nonprofits List 2025 is also available as a website, where you can sort by mission, funding needed or confidence, or do a search and have handy buttons.

Table of Contents

Organizations where I have the highest confidence in straightforward modest donations now, if your goals and model of the world align with theirs, are in bold, for those who don’t want to do a deep dive.

A Word of Warning

The SFF recommender process is highly time constrained, and in general I am highly time constrained.

Even though I used well beyond the number of required hours in both 2024 and 2025, there was no way to do a serious investigation of all the potentially exciting applications. Substantial reliance on heuristics was inevitable.

Also your priorities, opinions, and world model could be very different from mine.

If you are considering donating a substantial (to you) amount of money, please do the level of personal research and consideration commensurate with the amount of money you want to give away.

If you are considering donating a small (to you) amount of money, or if the requirement to do personal research might mean you don’t donate to anyone at all, I caution the opposite: Only do the amount of optimization and verification and such that is worth its opportunity cost. Do not let the perfect be the enemy of the good.

For more details of how the SFF recommender process works, see my post on the process.

Note that donations to some of the organizations below may not be tax deductible.

A Note To Charities

I apologize in advance for any errors, any out of date information, and for anyone who I included who I did not realize would not want to be included. I did my best to verify information, and to remove any organizations that do not wish to be included.

If you wish me to issue a correction of any kind, or to update your information, I will be happy to do that at least through the end of the year.

If you wish me to remove your organization entirely, for any reason, I will do that, too.

What I unfortunately cannot do, in most cases, is take the time to analyze or debate beyond that. I also can’t consider additional organizations for inclusion. My apologies.

The same is true for the website version.

I am giving my full opinion on all organizations listed, but where I feel an organization would be a poor choice for marginal dollars even within its own cause and intervention area, or I anticipate my full opinion would not net help them, they are silently not listed.

Use Your Personal Theory of Impact

Listen to arguments and evidence. But do not let me, or anyone else, tell you any of:

What is important.
What is a good cause.
What types of actions are best to make the change you want to see in the world.
What particular strategies are most promising.
That you have to choose according to some formula or you’re an awful person.

This is especially true when it comes to policy advocacy, and especially in AI.

If an organization is advocating for what you think is bad policy, or acting in a way that does bad things, don’t fund them!

If an organization is advocating or acting in a way you think is ineffective, don’t fund them!

Only fund people you think advance good changes in effective ways.

Not cases where I think that. Cases where you think that.

During SFF, I once again in 2025 chose to deprioritize all meta-level activities and talent development. I see lots of good object-level work available to do, and I expected others to often prioritize talent and meta activities.

The counterargument to this is that quite a lot of money is potentially going to be freed up soon as employees of OpenAI and Anthropic gain liquidity, including access to DAFs (donor advised funds). This makes expanding the pool more exciting.

I remain primarily focused on those who in some form were helping ensure AI does not kill everyone. I continue to see highest value in organizations that influence lab or government AI policies in the right ways, and continue to value Agent Foundations style and other off-paradigm technical research approaches.

Use Your Local Knowledge

I believe that the best places to give are the places where you have local knowledge.

If you know of people doing great work or who could do great work, based on your own information, then you can fund and provide social proof for what others cannot.

The less legible to others the cause, and the harder it is to fit it into the mission statements and formulas of various big donors, the more excited you should be to step forward, if the cause is indeed legible to you. This keeps you grounded, helps others find the show (as Tyler Cowen says), is more likely to be counterfactual funding, and avoids information cascades or looking under streetlights for the keys.

Most importantly it avoids adverse selection. The best legible opportunities for funding, the slam dunk choices? Those are probably getting funded. The legible things that are left are the ones that others didn’t sufficiently fund yet.

If you know why others haven’t funded, because they don’t know about the opportunity? That’s a great trade.

Unconditional Grants to Worthy Individuals Are Great

The process of applying for grants, raising money, and justifying your existence sucks.

A lot.

It especially sucks for many of the creatives and nerds that do a lot of the best work.

It also sucks to have to worry about running out of money, or to have to plan your work around the next time you have to justify your existence, or to be unable to be confident in choosing ambitious projects.

If you have to periodically go through this process, and are forced to continuously worry about making your work legible and how others will judge it, that will substantially hurt your true productivity. At best it is a constant distraction. By default, it is a severe warping effect. A version of this phenomenon is doing huge damage to academic science.

As I noted in my AI updates, the reason this blog exists is that I received generous, essentially unconditional, anonymous support to ‘be a public intellectual’ and otherwise pursue whatever I think is best. My benefactors offer their opinions when we talk because I value their opinions, but they never try to influence my decisions, and I feel zero pressure to make my work legible in order to secure future funding.

If you have money to give, and you know individuals who should clearly be left to do whatever they think is best without worrying about raising or earning money, who you are confident would take advantage of that opportunity and try to do something great, then giving them unconditional grants is a great use of funds, including giving them ‘don’t worry about reasonable expenses’ levels of funding.

This is especially true when combined with ‘retrospective funding,’ based on what they have already done. It would be great if we established a tradition and expectation that people who make big contributions can expect such rewards.

Not as unconditionally, it’s also great to fund specific actions and projects and so on that you see not happening purely through lack of money, especially when no one is asking you for money.

This includes things that you want to exist, but that don’t have a path to sustainability or revenue, or would be importantly tainted if they needed to seek that. Fund the project you want to see in the world. This can also be purely selfish, often in order to have something yourself you need to create it for everyone, and if you’re tempted there’s a good chance that’s a great value.

Do Not Think Only On the Margin, and Also Use Decision Theory

Resist the temptation to think purely on the margin, asking only what one more dollar can do. The incentives get perverse quickly. Organizations are rewarded for putting their highest impact activities in peril. Organizations that can ‘run lean’ or protect their core activities get punished.

If you always insist on being a ‘funder of last resort’ that requires key projects or the whole organization otherwise be in trouble, you’re defecting. Stop it.

Also, you want to do some amount of retrospective funding. If people have done exceptional work in the past, you should be willing to give them a bunch more rope in the future, above and beyond the expected value of their new project.

Don’t make everyone constantly reprove their cost effectiveness each year, or at least give them a break. If someone has earned your trust, then if this is the project they want to do next, presume they did so because of reasons, although you are free to disagree with those reasons.

Compare Notes With Those Individuals You Trust

This especially goes for AI lab employees. There’s no need for everyone to do all of their own research, you can and should compare notes with those who you can trust, and this is especially great when they’re people you know well.

What I do worry about is too much outsourcing of decisions to larger organizations and institutional structures, including those of Effective Altruism but also others, or letting your money go directly to large foundations where it will often get captured.

Beware Becoming a Fundraising Target

Jaan Tallinn created SFF in large part to intentionally take his donation decisions out of his hands, so he could credibly tell people those decisions were out of his hands, so he would not have to constantly worry that people he talked to were attempting to fundraise.

This is a huge deal. Communication, social life and a healthy information environment can all be put in danger by this.

And the Nominees Are

Time to talk about the organizations themselves.

Rather than offer precise rankings, I divided by cause category and into three confidence levels.

High confidence means I have enough information to be confident the organization is at least a good pick.
Medium or low confidence means exactly that – I have less confidence that the choice is wise, and you should give more consideration to doing your own research.
If my last investigation was in 2024, and I haven’t heard anything, I will have somewhat lower confidence now purely because my information is out of date.

Low confidence is still high praise, and very much a positive assessment! Most organizations would come nowhere close to making the post at all.

If an organization is not listed, that does not mean I think they would be a bad pick – they could have asked not to be included, or I could be unaware of them or their value, or I could simply not have enough confidence to list them.

I know how Bayesian evidence works, but this post is not intended as a knock on anyone, in any way. Some organizations that are not here would doubtless have been included, if I’d had more time.

I try to give a sense of how much detailed investigation and verification I was able to complete, and what parts I have confidence in versus not. Again, my lack of confidence will often be purely about my lack of time to get that confidence.

Unless I already knew them from elsewhere, assume no organizations here got as much attention as they deserve before you decide on what for you is a large donation.

I’m tiering based on how I think about donations from you, from outside SFF.

I think the regranting organizations were clearly wrong choices from within SFF, but are reasonable picks if you don’t want to do extensive research, especially if you are giving small.

In terms of funding levels needed, I will similarly divide into three categories.

They roughly mean this, to the best of my knowledge:

Low: Could likely be fully funded with less than ~$250k.

Medium: Could plausibly be fully funded with between ~$250k and ~$2 million.

High: Could probably make good use of more than ~$2 million.

These numbers may be obsolete by the time you read this. If you’re giving a large amount relative to what they might need, check with the organization first, but also do not be so afraid of modest amounts of ‘overfunding’ as relieving fundraising pressure is valuable and as I noted it is important not to only think on the margin.

A lot of organizations are scaling up rapidly, looking to spend far more money than they have in the past. This was true in 2024, and 2025 has only accelerated this trend. A lot more organizations are in ‘High’ now but I decided not to update the thresholds.

Everyone seems eager to double their headcount. I’m not putting people into the High category unless I am confident they can scalably absorb more funding after SFF.

The person who I list as the leader of an organization will sometimes accidentally be whoever was in charge of fundraising rather than strictly the leader. Partly the reason for listing it is to give context and some of you can go ‘oh right, I know who that is,’ and the other reason is that all organization names are often highly confusing – adding the name of the organization’s leader allows you a safety check, to confirm that you are indeed pondering the same organization I am thinking of!

Organizations that Are Literally Me

This is my post, so I get to list Balsa Research first. (I make the rules here.)

If that’s not what you’re interested in, you can of course skip the section.

Balsa Research

Focus: Groundwork starting with studies to allow repeal of the Jones Act

Leader: Zvi Mowshowitz

Funding Needed: Medium

Confidence Level: High

Our first target continues to be the Jones Act. With everything happening in 2025, it is easy to get distracted. We have decided to keep eyes on the prize.

We’ve commissioned two studies. Part of our plan is to do more of them, and also do things like draft model repeals and explore ways to assemble a coalition and to sell and spread the results, to enable us to have a chance at repeal.

We also are networking, gathering information, publishing findings where there are information holes or where we can offer superior presentations, planning possible collaborations, and responding quickly in case of a crisis in related areas. We believe we meaningfully reduced the probability that certain very damaging additional maritime regulations could have become law, as described in this post.

Other planned cause areas include NEPA reform and federal housing policy (to build more housing where people want to live).

We have one full time worker on the case and are trying out a potential second one.

I don’t intend to have Balsa work on AI or assist with my other work, or to take personal compensation, unless I get substantially larger donations than we have had previously, that are either dedicated to those purposes or that at least come with the explicit understanding I should consider doing that.

Further donations would otherwise be for general support.

The pitch for Balsa, and the reason I am doing it, is in two parts.

I believe Jones Act repeal and many other abundance agenda items are neglected, tractable and important, and that my way of focusing on what matters can advance them. That the basic work that needs doing is not being done, it would be remarkably cheap to do a lot of it and do it well, and that this would give us a real if unlikely chance to get a huge win if circumstances break right. Chances for progress currently look grim, but winds can change quickly, we need to be ready, and also we need to stand ready to mitigate the chance things get even worse.

I also believe that if people do not have hope for the future, do not have something to protect and fight for, or do not think good outcomes are possible, that people won’t care about protecting the future. And that would be very bad, because we are going to need to fight to protect our future if we want to have one, or have a good one.

You got to give them hope.

I could go on, but I’ll stop there.

Donate here, or get in touch at donations@balsaresearch.com.

Don’t Worry About the Vase

Focus: Zvi Mowshowitz writes a lot of words, really quite a lot.

Leader: Zvi Mowshowitz

Funding Needed: None, but it all helps, could plausibly absorb a lot

Confidence Level: High

You can also of course always donate directly to my favorite charity.

By which I mean me. I always appreciate your support, however large or small.

The easiest way to help on a small scale (of course) is a Substack subscription or Patreon. Paid substack subscriptions punch above their weight because they assist with the sorting algorithm, and also for their impact on morale.

If you want to go large then reach out to me.

Thanks to generous anonymous donors, I am able to write full time and mostly not worry about money. That is what makes this blog possible.

I want to as always be 100% clear: I am totally, completely fine as is, as is the blog.

Please feel zero pressure here, as noted throughout there are many excellent donation opportunities out there.

Additional funds are still welcome. There are levels of funding beyond not worrying.

Such additional support is always highly motivating.

Also there are absolutely additional things I could and would throw money at to improve the blog, potentially including hiring various forms of help or even expanding to more of a full news operation or startup.

Organizations Focusing On AI Non-Technical Research and Education

As a broad category, these are organizations trying to figure things out regarding AI existential risk, without centrally attempting to either do technical work or directly to influence policy and discourse.

Lightcone Infrastructure is my current top pick across all categories. If you asked me where to give a dollar, or quite a few dollars, to someone who is not me, I would tell you to fund Lightcone Infrastructure.

Lightcone Infrastructure

Focus: Rationality community infrastructure, LessWrong, the Alignment Forum and Lighthaven.

Leaders: Oliver Habryka, Raymond Arnold, Ben Pace

Funding Needed: High

Confidence Level: High

Disclaimer: I am on the CFAR board which used to be the umbrella organization for Lightcone and still has some lingering ties. My writing appears on LessWrong. I have long time relationships with everyone involved. I have been to several reliably great workshops or conferences at their campus at Lighthaven. So I am conflicted here.

With that said, Lightcone is my clear number one. I think they are doing great work, both in terms of LessWrong and also Lighthaven. There is the potential, with greater funding, to enrich both of these tasks, and also for expansion.

There is a large force multiplier here (although that is true of a number of other organizations I list as well).

They made their 2024 fundraising pitch here, I encourage reading it.

Where I am beyond confident is that if LessWrong, the Alignment Forum or the venue Lighthaven were unable to continue, any one of these would be a major, quite bad unforced error.

LessWrong and the Alignment Forum a central part of the infrastructure of the meaningful internet.

Lighthaven is miles and miles away the best event venue I have ever seen. I do not know how to convey how much the design contributes to having a valuable conference, designed to facilitate the best kinds of conversations via a wide array of nooks and pathways designed with the principles of Christopher Alexander. This contributes to and takes advantage of the consistently fantastic set of people I encounter there.

The marginal costs here are large (~$3 million per year, some of which is made up by venue revenue), but the impact here is many times that, and I believe they can take on more than ten times that amount and generate excellent returns.

If we can go beyond short term funding needs, they can pay off the mortgage to secure a buffer, and buy up surrounding buildings to secure against neighbors (who can, given this is Berkeley, cause a lot of trouble) and to secure more housing and other space. This would secure the future of the space.

I would love to see them then expand into additional spaces. They note this would also require the right people.

Donate through every.org, or contact team@lesswrong.com.

The AI Futures Project

Focus: AI forecasting research projects, governance research projects, and policy engagement, in that order.

Leader: Daniel Kokotajlo, with Eli Lifland

Funding Needed: None Right Now

Confidence Level: High

Of all the ‘shut up and take my money’ applications in the 2024 round where I didn’t have a conflict of interest, even before I got to participate in their tabletop wargame exercise, I judged this the most ‘shut up and take my money’-ist. At The Curve, I got to participate in the exercise and participate in discussions around it, I’ve since done several more, and I’m now even more confident this is an excellent pick.

I continue to think it is a super strong case for retroactive funding as well. Daniel walked away from OpenAI, and what looked to be most of his net worth, to preserve his right to speak up. That led to us finally allowing others at OpenAI to speak up as well.

This is how he wants to speak up, and try to influence what is to come, based on what he knows. I don’t know if it would have been my move, but the move makes a lot of sense, and it has already paid off big. AI 2027 was read by the Vice President, who took it seriously, along with many others, and greatly informed the conversation. I believe the discourse is much improved as a result, and the possibility space has improved.

Note that they are comfortably funded through the medium term via private donations and their recent SFF grant.

Donate through every.org, or contact Jonas Vollmer.

Effective Institutions Project (EIP) (For Their Flagship Initiatives)

Focus: AI governance, advisory and research, finding how to change decision points

Leader: Ian David Moss

Funding Needed: Medium

Confidence Level: High

EIP operates on two tracks. They have their flagship initiatives and attempts to intervene directly. They also serve as donation advisors, which I discuss in that section.

Their current flagship initiative plans are to focus on the intersection of AI governance and the broader political and economic environment, especially risks of concentration of power and unintentional power shifts from humans to AIs.

Can they indeed identify ways to target key decision points, and make a big difference? One can look at their track record. I’ve been asked to keep details confidential, but based on my assessment of private information, I confirmed they’ve scored some big wins including that they helped improve safety practices at a major AI lab, and will plausibly continue to be able to have high leverage and punch above their funding weight. You can read about some of the stuff that they can talk about here in a Founders Pledge write up.

It seems important that they be able to continue their work on all this.

I also note that in SFF I allocated less funding to EIP than I would in hindsight have liked to allocate, due to quirks about the way matching funds worked and my attempts to adjust my curves to account for it.

Donate through every.org, or contact info@effectiveinstitutionsproject.org.

Artificial Intelligence Policy Institute (AIPI)

Focus: Primarily polls about AI, also lobbying and preparing for crisis response.

Leader: Daniel Colson.

Also Involved: Mark Beall and Daniel Eth

Funding Needed: High

Confidence Level: High

Those polls about how the public thinks about AI, including several from last year around SB 1047 including an adversarial collaboration with Dean Ball?

Remarkably often, these are the people that did that. Without them, few would be asking those questions. Ensuring that someone is asking is super helpful. With some earlier polls I was a bit worried that the wording was slanted, and that will always be a concern with a motivated pollster, but I think recent polls have been much better at this, and they are as close to neutral as one can reasonably expect.

There are those who correctly point out that even now in 2025 the public’s opinions are weakly held and low salience, and that all you’re often picking up is ‘the public does not like AI and it likes regulation.’

Fair enough. Someone still has to show this, and show it applies here, and put a lie to people claiming the public goes the other way, and measure how things change over time. We need to be on top of what the public is thinking, including to guard against the places it wants to do dumb interventions.

They don’t only do polling. They also do lobbying and prepare for crisis responses.

Donate here, or use their contact form to get in touch.

AI Lab Watch

Focus: Monitoring the AI safety record and plans of the frontier AI labs

Leader: Zach Stein-Perlman

Funding Needed: Low

Confidence Level: High

Zach has consistently been one of those on top of the safety and security plans, the model cards and other actions of the major labs, both writing up detailed feedback from a skeptical perspective and also compiling the website and its scores in various domains. Zach is definitely in the ‘demand high standards that would actually work and treat everything with skepticism’ school of all this, which I feel is appropriate, and I’ve gotten substantial benefit of his work several times.

However, due to uncertainty about whether this is the best thing for him to work on, and thus not being confident he will have this ball, Zach is not currently accepting funding, but would like people who are interested in donations to contact him via Intercom on the AI Lab Watch website.

Palisade Research

Focus: AI capabilities demonstrations to inform decision makers on capabilities and loss of control risks

Leader: Jeffrey Ladish

Funding Needed: High

Confidence Level: High

This is clearly an understudied approach. People need concrete demonstrations. Every time I get to talking with people in national security or otherwise get closer to decision makers who aren’t deeply into AI and in particular into AI safety concerns, you need to be as concrete and specific as possible – that’s why I wrote Danger, AI Scientist, Danger the way I did. We keep getting rather on-the-nose fire alarms, but it would be better if we could get demonstrations even more on the nose, and get them sooner, and in a more accessible way.

Since last time, I’ve had a chance to see their demonstrations in action several times, and I’ve come away feeling that they have mattered.

I have confidence that Jeffrey is a good person to continue to put this plan into action.

To donate, click here or email donate@palisaderesearch.org.

CivAI

Focus: Visceral demos of AI risks

Leader: Sid Hiregowdara

Funding Needed: High

Confidence Level: Medium

I was impressed by the demo I was given (so a demo demo?). There’s no question such demos fill a niche and there aren’t many good other candidates for the niche.

The bear case is that the demos are about near term threats, so does this help with the things that matter? It’s a good question. My presumption is yes, that raising situational awareness about current threats is highly useful. That once people notice that there is danger, that they will ask better questions, and keep going. But I always do worry about drawing eyes to the wrong prize.

To donate, click here or email contact@civai.org.

AI Safety Info (Robert Miles)

Focus: Making YouTube videos about AI safety, starring Rob Miles

Leader: Rob Miles

Funding Needed: Low

Confidence Level: High

I think these are pretty great videos in general, and given what it costs to produce them we should absolutely be buying their production. If there is a catch, it is that I am very much not the target audience, so you should not rely too much on my judgment of what is and isn’t effective video communication on this front, and you should confirm you like the cost per view.

To donate, join his patreon or contact him at robertmilesai@robertskmiles.com.

Intelligence Rising

Focus: Facilitation of the AI scenario roleplaying exercises including Intelligence Rising

Leader: Shahar Avin

Funding Needed: Low

Confidence Level: High

I haven’t had the opportunity to play Intelligence Rising, but I have read the rules to it, and heard a number of strong after action reports (AARs). They offered this summary of insights in 2024. The game is clearly solid, and it would be good if they continue to offer this experience and if more decision makers play it, in addition to the AI Futures Project TTX.

To donate, reach out to team@intelligencerising.org.

Convergence Analysis

Focus: A series of sociotechnical reports on key AI scenarios, governance recommendations and conducting AI awareness efforts.

Leader: David Kristoffersson

Funding Needed: High (combining all tracks)

Confidence Level: Low

They do a variety of AI safety related things. Their Scenario Planning continues to be what I find most exciting, although I’m also somewhat interested in their modeling cooperation initiative as well. It’s not as neglected as it was a year ago, but we could definitely use more work than we’re getting. For track record you check out their reports from 2024 in this area, and see if you think that was good work, and the rest of their website has more.

Their donation page is here, or you can contact contact@convergenceanalysis.org.

IASEAI (International Association for Safe and Ethical Artificial Intelligence)

Focus: Grab bag of AI safety actions, research, policy, community, conferences, standards

Leader: Mark Nitzberg

Funding Needed: High

Confidence Level: Low

There are some clearly good things within the grab bag, including some good conferences and it seems substantial support for Geoffrey Hinton, but for logistical reasons I didn’t do a close investigation to see if the overall package looked promising. I’m passing the opportunity along.

Donate here, or contact them at info@iaseai.org.

The AI Whistleblower Initiative

Focus: Whistleblower advising and resources for those in AI labs warning about catastrophic risks, including via Third Opinion.

Leader: Larl Koch

Funding Needed: High

Confidence Level: Medium

I’ve given them advice, and at least some amount of such resourcing is obviously highly valuable. We certainly should be funding Third Opinion, so that if someone wants to blow the whistle they can have help doing it. The question is whether if it scales this loses its focus.

Donate here, or reach out to hello@aiwi.org.

Organizations Related To Potentially Pausing AI Or Otherwise Having A Strong International AI Treaty Pause AI and Pause AI Global

Focus: Advocating for a pause on AI, including via in-person protests

Leader: Holly Elmore (USA) and Joep Meindertsma (Global)

Funding Level: Low

Confidence Level: Medium

Some people say that those who believe we should pause AI would be better off staying quiet about it, rather than making everyone look foolish.

I disagree.

I don’t think pausing right now is a good idea. I think we should be working on the transparency, state capacity, technical ability and diplomatic groundwork to enable a pause in case we need one, but that it is too early to actually try to implement one.

But I do think that if you believe we should pause? Then you should say that we should pause. I very much appreciate people standing up, entering the arena and saying what they believe in, including quite often in my comments. Let the others mock all they want.

If you agree with Pause AI that the right move is to Pause AI, and you don’t have strong strategic disagreements with their approach, then you should likely be excited to fund this. If you disagree, you have better options.

Either way, they are doing what they, given their beliefs, should be doing.

Donate here, or reach out to joep@pauseai.info.

MIRI

Focus: At this point, primarily AI policy advocacy, letting everyone know that If Anyone Builds It, Everyone Dies and all that, plus some research

Leaders: Malo Bourgon, Eliezer Yudkowsky

Funding Needed: High

Confidence Level: High

MIRI, concluding that it is highly unlikely alignment will make progress rapidly enough otherwise, has shifted its strategy to largely advocate for major governments coming up with an international agreement to halt AI progress and to do communications, although research still looks to be a large portion of the budget, and they have dissolved its agent foundations team. Hence the book.

That is not a good sign for the world, but it does reflect their beliefs.

They have accomplished a lot. The book is at least a modest success on its own terms in moving things forward.

I strongly believe they should be funded to continue to fight for a better future however they think is best, even when I disagree with their approach.

This is very much a case of ‘do this if and only if this aligns with your model and preferences.’

Donate here, or reach out to harlan@intelligence.org.

Existential Risk Observatory

Focus: Pause-relevant research

Leader: Otto Barten

Funding Needed: Low

Confidence Level: Medium

Mostly this is the personal efforts of Otto Barten, ultimately advocating for a conditional pause. For modest amounts of money, in prior years he’s managed to have a hand in some high profile existential risk events and get the first x-risk related post into TIME magazine. He’s now pivoted to pause-relevant research (as in how to implement one via treaties, off switches, evals and threat models).

The track record and my prior investigation is less relevant now, so I’ve bumped them down to low confidence, but it would definitely be good to have the technical ability to pause and not enough work is being done on that.

To donate, click here, or get in touch at info@existentialriskobservatory.org.

Organizations Focusing Primary On AI Policy and Diplomacy

Some of these organizations also look at bio policy or other factors, but I judge those here as being primarily concerned with AI.

In this area, I am especially keen to rely on people with good track records, who have shown that they can build and use connections and cause real movement. It’s so hard to tell what is and isn’t effective, otherwise. Often small groups can pack a big punch, if they know where to go, or big ones can be largely wasted – I think that most think tanks on most topics are mostly wasted even if you believe in their cause.

Center for AI Safety and the CAIS Action Fund

Focus: AI research, field building and advocacy

Leaders: Dan Hendrycks

Funding Needed: High

Confidence Level: High

They did the CAIS Statement on AI Risk, helped SB 1047 get as far as it did, and have improved things in many other ways. Some of these other ways are non-public. Some of those non-public things are things I know about and some aren’t. I will simply say the counterfactual policy world is a lot worse. They’ve clearly been punching well above their weight in the advocacy space. The other arms are no slouch either, lots of great work here. Their meaningful rolodex and degree of access is very strong and comes with important insight into what matters.

They take a lot of big swings and aren’t afraid of taking risks or looking foolish. I appreciate that, even when a given attempt doesn’t fully work.

If you want to focus on their policy, then you can fund their 501(c)(4), the Action Fund, since 501c(3)s are limited in how much they can spend on political activities, keeping in mind the tax implications of that. If you don’t face any tax implications I would focus first on the 501(c)(4).

We should definitely find a way to fund at least their core activities.

Donate to the Action Fund for funding political activities, or the 501(c)(3) for research. They can be contacted at contact@safe.ai.

Foundation for American Innovation (FAI)

Focus: Tech policy research, thought leadership, educational outreach to government, fellowships.

Leader: Grace Meyer

Funding Needed: High

Confidence Level: High

FAI is centrally about innovation. Innovation is good, actually, in almost all contexts, as is building things and letting people do things.

AI is where this gets tricky. People ‘supporting innovation’ are often using that as an argument against all regulation of AI, and indeed I am dismayed to see so many push so hard on this exactly in the one place I think they are deeply wrong, when we could work together on innovation (and abundance) almost anywhere else.

FAI and resident AI studiers Samuel Hammond and Dean Ball are in an especially tough spot, because they are trying to influence AI policy from the right and not get expelled from that coalition or such spaces. There’s a reason we don’t have good alternative options for this. That requires striking a balance.

I’ve definitely had my disagreements with Hammond, including strong disagreements with his 95 theses on AI although I agreed far more than I disagreed, and I had many disagreements with his AI and Leviathan as well. He’s talked on the Hill about ‘open model diplomacy.’

I’ve certainly had many strong disagreements with Dean Ball as well, both in substance and rhetoric. Sometimes he’s the voice of reason and careful analysis, other times (from my perspective) he can be infuriating, most recently in discussions of the Superintelligence Statement, remarkably often he does some of both in the same post. He was perhaps the most important opposer of SB 1047 and went on to a stint at the White House before joining FAI.

Yet here is FAI, rather high on the list. They’re a unique opportunity, you go to war with the army you have, and both Ball and Hammond have stuck their neck out in key situations. Hammond came out opposing the moratorium. They’ve been especially strong on compute governance.

I have private reasons to believe that FAI has been effective and we can expect that to continue, and its other initiatives also mostly seem good. We don’t have to agree on everything else, so long as we all want good things and are trying to figure things out, and I’m confident that is the case here.

I am especially excited that they can speak to the Republican side of the aisle in the R’s native language, which is difficult for most in this space to do.

An obvious caveat is that if you are not interested in the non-AI pro-innovation part of the agenda (I certainly approve, but it’s not obviously a high funding priority for most readers) then you’ll want to ensure it goes where you want it.

To donate, click here, or contact them using the form here.

Encode AI (Formerly Encode Justice)

Focus: Youth activism on AI safety issues

Leader: Sneha Revanur

Funding Needed: Medium

Confidence Level: High

They started out doing quite a lot on a shoestring budget by using volunteers, helping with SB 1047 and in several other places. Now they are turning pro, and would like to not be on a shoestring. I think they have clearly earned that right. The caveat is risk of ideological capture. Youth organizations tend to turn to left wing causes.

The risk here is that this effectively turns mostly to AI ethics concerns. It’s great that they’re coming at this without having gone through the standard existential risk ecosystem, but that also heightens the ideological risk.

I continue to believe it is worth the risk.

To donate, go here. They can be contacted at sneha@encodeai.org.

The Future Society

Focus: AI governance standards and policy.

Leader: Nicolas Moës

Funding Needed: High

Confidence Level: High

I’ve seen credible sources saying they do good work, and that they substantially helped orient the EU AI Act to at least care at all about frontier general AI. The EU AI Act was not a good bill, but it could easily have been a far worse one, doing much to hurt AI development while providing almost nothing useful for safety.

We should do our best to get some positive benefits out of the whole thing. And indeed, they helped substantially improve the EU Code of Practice, which was in hindsight remarkably neglected otherwise.

They’re also active around the world, including the USA and China.

Donate here, or contact them here.

Safer AI

Focus: Specifications for good AI safety, also directly impacting EU AI policy

Leader: Henry Papadatos

Funding Needed: Medium

Confidence Level: Low

I’ve been impressed by Simeon and his track record, including here. Simeon is stepping down as leader to start a company, which happened post-SFF, so they would need to be reevaluated in light of this before any substantial donation.

Donate here, or contact them at contact@safer-ai.org.

Institute for AI Policy and Strategy (IAPS)

Focus: Papers and projects for ‘serious’ government circles, meetings with same, policy research

Leader: Peter Wildeford

Funding Needed: Medium

Confidence Level: High

I have a lot of respect for Peter Wildeford, and they’ve clearly put in good work and have solid connections down, including on the Republican side where better coverage is badly needed, and the only other solid lead we have is FAI. Peter has also increasingly been doing strong work directly via Substack and Twitter that has been helpful to me and that I can observe directly. They are strong on hardware governance and chips in particular (as is FAI).

Given their goals and approach, funding from outside the traditional ecosystem sources would be extra helpful, ideally such efforts are fully distinct from OpenPhil.

With the shifting landscape and what I’ve observed, I’m moving them up to high confidence and priority.

Donate here, or contact them at jmarron@iaps.ai.

AI Standards Lab (Holtman Research)

Focus: Accelerating the writing of AI safety standards

Leaders: Koen Holtman and Chin Ze Shen

Funding Needed: Medium

Confidence Level: High

They help facilitate the writing of AI safety standards, for EU/UK/USA, including on the recent EU Code of Practice. They have successfully gotten some of their work officially incorporated, and another recommender with a standards background was impressed by the work and team.

This is one of the many things that someone has to do, and where if you step up and do it and no one else does that can go pretty great. Having now been involved in bill minutia myself, I know it is thankless work, and that it can really matter, both for public and private standards, and they plan to pivot somewhat to private standards.

I’m raising my confidence to high that this is at least a good pick, if you want to fund the writing of standards.

To donate, go here or reach out to hello@aistandardslab.org.

Safe AI Forum

Focus: International AI safety conferences

Leader: Fynn Heide and Sophie Thomson

Funding Needed: Medium

Confidence Level: Low

They run the IDAIS series of conferences, including successful ones involving China. I do wish I had a better model of what makes such a conference actually matter versus not mattering, but these sure seem like they should matter, and certainly well worth their costs to run them.

To donate, contact them using the form at the bottom of the page here.

Center For Long Term Resilience

Focus: UK Policy Think Tank focusing on ‘extreme AI risk and biorisk policy.’

Leader: Angus Mercer

Funding Needed: High

Confidence Level: Low

The UK has shown promise in its willingness to shift its AI regulatory focus to frontier models in particular. It is hard to know how much of that shift to attribute to any particular source, or otherwise measure how much impact there has been or might be on final policy.

They have endorsements of their influence from philosopher Toby Ord, Former Special Adviser to the UK Prime Minister Logan Graham, and Senior Policy Adviser Nitarshan Rajkumar.

I reached out to a source with experience in the UK government who I trust, and they reported back they are a fan and pointed to some good things they’ve helped with. There was a general consensus that they do good work, and those who investigated where impressed.

However, I have concerns. Their funding needs are high, and they are competing against many others in the policy space, many of which have very strong cases. I also worry their policy asks are too moderate, which might be an advantage for others.

My lower confidence this year is a combination of worries about moderate asks, worry about organizational size, and worries about the shift in governments in the UK and the UK’s ability to have real impact elsewhere. But if you buy the central idea of this type of lobbying through the UK and are fine with a large budget, go for it.

Donate here, or reach out to info@longtermresilience.org.

Simon Institute for Longterm Governance

Focus: Foundations and demand for international cooperation on AI governance and differential tech development

Leader: Konrad Seifert and Maxime Stauffer

Funding Needed: High

Confidence Level: Low

As with all things diplomacy, hard to tell the difference between a lot of talk and things that are actually useful. Things often look the same either way for a long time. A lot of their focus is on the UN, so update either way based on how useful you think that approach is, and also that makes it even harder to get a good read.

They previously had a focus on the Global South and are pivoting to China, which seems like a more important focus.

To donate, scroll down on this page to access their donation form, or contact them at contact@simoninstitute.ch.

Legal Advocacy for Safe Science and Technology

Focus: Legal team for lawsuits on catastrophic risk and to defend whistleblowers.

Leader: Tyler Whitmer

Funding Needed: Medium

Confidence Level: Medium

I wasn’t sure where to put them, but I suppose lawsuits are kind of policy by other means in this context, or close enough?

I buy the core idea of having a legal team on standby for catastrophic risk related legal action in case things get real quickly is a good idea, and I haven’t heard anyone else propose this, although I do not feel qualified to vet the operation. They were one of the organizers of the NotForPrivateGain.org campaign against the OpenAI restructuring.

I definitely buy the idea of an AI Safety Whistleblower Defense Fund, which they are also doing. Knowing there will be someone to step up and help if it comes to that changes the dynamics in helpful ways.

Donors who are interested in making relatively substantial donations or grants should contact tyler@lasst.org, for smaller amounts click here.

Institute for Law and AI

Focus: Legal research on US/EU law on transformational AI, fellowships, talent

Leader: Moritz von Knebel

Involved: Gabe Weil

Funding Needed: High

Confidence Level: Low

I’m confident that they should be funded at all, the question is if this should be scaled up quite a lot, and what aspects of this would scale in what ways. If you can be convinced that the scaling plans are worthwhile this could justify a sizable donation.

Donate here, or contact them at hello@law-ai.org.

Macrostrategy Research Institute

Focus: Amplify Nick Bostrom

Leader: Toby Newberry

Funding Needed: High

Confidence Level: Low

If you think Nick Bostrom is doing great work and want him to be more effective, then this is a way to amplify that work. In general, ‘give top people support systems’ seems like a good idea that is underexplored.

Get in touch at toby.newberry@gmail.com.

Secure AI Project

Focus: Advocacy for public safety and security protocols (SSPs) and related precautions

Leader: Nick Beckstead

Funding Needed: High

Confidence Level: High

I’ve had the opportunity to consult and collaborate with them and I’ve been consistently impressed. They’re the real deal, they pay attention to detail and care about making it work for everyone, and they’ve got results. I’m a big fan.

Donate here, or contact them at info@secureaiproject.org.

Organizations Doing ML Alignment Research

This category should be self-explanatory. Unfortunately, a lot of good alignment work still requires charitable funding. The good news is that (even more than last year when I wrote the rest of this introduction) there is a lot more funding, and willingness to fund, than there used to be, and also the projects generally look more promising.

The great thing about interpretability is that you can be confident you are dealing with something real. The not as great thing is that this can draw too much attention to interpretability, and that you can fool yourself into thinking that All You Need is Interpretability.

The good news is that several solid places can clearly take large checks.

I didn’t investigate too deeply on top of my existing knowledge here in 2024, because at SFF I had limited funds and decided that direct research support wasn’t a high enough priority, partly due to it being sufficiently legible.

We should be able to find money previously on the sidelines eager to take on many of these opportunities. Lab employees are especially well positioned, due to their experience and technical knowledge and connections, to evaluate such opportunities, and also to provide help with access and spreading the word.

Model Evaluation and Threat Research (METR)

Formerly ARC Evaluations.

Focus: Model evaluations

Leaders: Beth Barnes, Chris Painter

Funding Needed: High

Confidence Level: High

Originally I wrote that we hoped to be able to get large funding for METR via non-traditional sources. That happened last year, and METR got major funding. That’s great news. Alas, they once again have to hit the fundraising trail.

METR has proven to be the gold standard for outside evaluations of potentially dangerous frontier model capabilities, and has proven its value even more so in 2025.

We very much need these outside evaluations, and to give the labs every reason to use them and no excuse not to use them, and their information has been invaluable. In an ideal world the labs would be fully funding METR, but they’re not.

So this becomes a place where we can confidently invest quite a bit of capital, make a legible case for why it is a good idea, and know it will probably be well spent.

If you can direct fully ‘square’ ‘outside’ funds that need somewhere legible to go and are looking to go large? I love METR for that.

To donate, click here. They can be contacted at info@metr.org.

Alignment Research Center (ARC)

Focus: Theoretically motivated alignment work

Leader: Jacob Hilton

Funding Needed: Medium

Confidence Level: High

There’s a long track record of good work here, and Paul Christiano remained excited as of 2024. If you are looking to fund straight up alignment work and don’t have a particular person or small group in mind, this is certainly a safe bet to put additional funds to good use and attract good talent.

Donate here, or reach out to jacob@alignment.org.

Apollo Research

Focus: Scheming, evaluations, and governance

Leader: Marius Hobbhahn

Funding Needed: Medium

Confidence Level: High

This is an excellent thing to focus on, and one of the places we are most likely to be able to show ‘fire alarms’ that make people sit up and notice. Their first year seems to have gone well, one example would be their presentation at the UK safety summit that LLMs can strategically deceive their primary users when put under pressure. They will need serious funding to fully do the job in front of them, hopefully like METR they can be helped by the task being highly legible.

They suggest looking at this paper, and also this one. I can verify that they are the real deal and doing the work.

To donate, reach out to info@apolloresearch.ai.

Cybersecurity Lab at University of Louisville

Focus: Support for Roman Yampolskiy’s lab and work

Leader: Roman Yampolskiy

Funding Needed: Low

Confidence Level: High

Roman Yampolskiy is the most pessimistic known voice about our chances of not dying from AI, and got that perspective on major platforms like Joe Rogan and Lex Fridman. He’s working on a book and wants to support PhD students.

Supporters can make a tax detectable gift to the University, specifying that they intend to fund Roman Yampolskiy and the Cyber Security lab.

Timaeus

Focus: Interpretability research

Leader:Jesse Hoogland, Daniel Murfet, Stan van Wingerden

Funding Needed: High

Confidence Level: High

Timaeus focuses on interpretability work and sharing their results. The set of advisors is excellent, including Davidad and Evan Hubinger. Evan, John Wentworth and Vanessa Kosoy have offered high praise, and there is evidence they have impacted top lab research agendas. They’re done what I think is solid work, although I am not so great at evaluating papers directly.

If you’re interested in directly funding interpretability research, that all makes this seem like a slam dunk. I’ve confirmed that this all continues to hold true in 2025.

To donate, get in touch with Jesse at jesse@timaeus.co. If this is the sort of work that you’re interested in doing, they also have a discord at http://devinterp.com/discord.

Simplex

Focus: Mechanistic interpretability of how inference breaks down

Leader: Paul Riechers and Adam Shai

Funding Needed: Medium

Confidence Level: High

I am not as high on them as I am on Timaeus, but they have given reliable indicators that they will do good interpretability work. I’d (still) feel comfortable backing them.

Donate here, or contact them via webform.

Far AI

Focus: Interpretability and other alignment research, incubator, hits based approach

Leader: Adam Gleave

Funding Needed: High

Confidence Level: Medium

They take the hits based approach to research, which is correct. I’ve gotten confirmation that they’re doing the real thing here. In an ideal world everyone doing the real thing would get supported, and they’re definitely still funding constrained.

To donate, click here. They can be contacted at hello@far.ai.

Alignment in Complex Systems Research Group

Focus: AI alignment research on hierarchical agents and multi-system interactions

Leader: Jan Kulveit

Funding Needed: Medium

Confidence Level: High

I liked ACS last year, and since then we’ve seen Gradual Disempowerment and other good work, which means this now falls into the category ‘this having funding problems would be an obvious mistake.’ I ranked them very highly in SFF, and there should be a bunch more funding room.

To donate, reach out to hello@epistea.org, and note that you are interested in donating to ACS specifically.

Apart Research

Focus: AI safety hackathons, MATS-style programs and AI safety horizon scanning.

Leaders: Esben Kran, Jason Schreiber

Funding Needed: Medium

Confidence Level: Low

I’m (still) confident in their execution of the hackathon idea, which was the central pitch at SFF although they inform me generally they’re more centrally into the MATS-style programs. My doubt for the hackathons is on the level of ‘is AI safety something that benefits from hackathons.’ Is this something one can, as it were, hack together usefully? Are the hackathons doing good counterfactual work? Or is this a way to flood the zone with more variations on the same ideas?

As with many orgs on the list, this one makes sense if and only if you buy the plan, and is one of those ‘I’m not excited but can see it being a good fit for someone else.’

To donate, click here. They can be reached at hello@apartresearch.com.

Transluce

Focus: Specialized superhuman systems for understanding and overseeing AI

Leaders: Jacob Steinhardt, Sarah Schwettmann

Funding Needed: High

Confidence Level: Medium

Last year they were a new org. Now they have now grown to 14 people and now have a solid track record and want to keep growing. I have confirmation the team is credible. The plan for scaling themselves is highly ambitious, with planned scale well beyond what SFF can fund. I haven’t done anything like the investigation into their plans and capabilities you would need before placing a bet that big, as AI research of all kinds gets expensive quickly.

If there is sufficient appetite to scale the amount of privately funded direct work of this type, then this seems like a fine place to look. I am optimistic on them finding interesting things, although on a technical level I am skeptical of the larger plan.

To donate, reach out to info@transluce.org.

Organizations Doing Other Technical Work AI Analysts @ RAND

Focus: Developing ‘AI analysts’ that can assist policy makers.

Leaders: John Coughlan

Funding Needed: High

Confidence Level: Medium

This is a thing that RAND should be doing and that should exist. There are obvious dangers here, but I don’t think this makes them substantially worse and I do think this can potentially improve policy a lot. RAND is well placed to get the resulting models to be actually used. That would enhance state capacity, potentially quite a bit.

The problem is that doing this is not cheap, and while funding this shouldn’t fall to those reading this, it plausibly does. This could be a good place to consider sinking quite a large check, if you believe in the agenda.

Donate here.

Organizations Doing Math, Decision Theory and Agent Foundations

Right now it looks likely that AGI will be based around large language models (LLMs). That doesn’t mean this is inevitable. I would like our chances better if we could base our ultimate AIs around a different architecture, one that was more compatible with being able to get it to do what we would like it to do.

One path for this is agent foundations, which involves solving math to make the programs work instead of relying on inscrutable giant matrices.

Even if we do not manage that, decision theory and game theory are potentially important for navigating the critical period in front of us, for life in general, and for figuring out what the post-transformation AI world might look like, and thus what choice we make now might do to impact that.

There are not that many people working on these problems. Actual Progress would be super valuable. So even if we expect the median outcome does not involve enough progress to matter, I think it’s still worth taking a shot.

The flip side is you worry about people ‘doing decision theory into the void’ where no one reads their papers or changes their actions. That’s a real issue. As is the increased urgency of other options. Still, I think these efforts are worth supporting, in general.

Orthogonal

Focus: AI alignment via agent foundations

Leaders: Tamsin Leake

Funding Needed: Medium

Confidence Level: High

I have funded Orthogonal in the past. They are definitely doing the kind of work that, if it succeeded, might actually amount to something, and would help us get through this to a future world we care about. It’s a long shot, but a long shot worth trying. They very much have the ‘old school’ Yudkowsky view that relatively hard takeoff is likely and most alignment approaches are fools errands. My sources are not as enthusiastic as they once were, but there are only a handful of groups trying that have any chance at all, and this still seems like one of them.

Donate here, or get in touch at tammy@orxl.org.

Topos Institute

Focus: Math for AI alignment

Leaders: Brendan Fong and David Spivak.

Funding Needed: High

Confidence Level: High

Topos is essentially Doing Math to try and figure out what to do about AI and AI Alignment. I’m very confident that they are qualified to (and actually will) turn donated money (partly via coffee) into math, in ways that might help a lot. I am also confident that the world should allow them to attempt this.

They’re now working with ARIA. That seems great.

Ultimately it all likely amounts to nothing, but the upside potential is high and the downside seems very low. I’ve helped fund them in the past and am happy about that.

To donate, go here, or get in touch at info@topos.institute.

Eisenstat Research

Focus: Two people doing research at MIRI, in particular Sam Eisenstat

Leader: Sam Eisenstat

Funding Needed: Medium

Confidence Level: High

Given Sam Eisenstat’s previous work, including from 2025, it seems worth continuing to support him, including supporting researchers. I still believe in this stuff being worth working on, obviously only support if you do as well. He’s funded for now but that’s still only limited runway.

To donate, contact sam@intelligence.org.

AFFINE Algorithm Design

Focus: Johannes Mayer does agent foundations work

Leader: Johannes Mayer

Funding Needed: Low

Confidence Level: Medium

Johannes Mayer does solid agent foundations work, and more funding would allow him to hire more help.

To donate, contact j.c.mayer240@gmail.com.

CORAL (Computational Rational Agents Laboratory)

Focus: Examining intelligence

Leader: Vanessa Kosoy

Funding Needed: Medium

Confidence Level: High

This is Vanessa Kosoy and Alex Appel, who have another research agenda formerly funded by MIRI that now needs to stand on its own after their refocus. I once again believe this work to be worth continuing even if the progress isn’t what one might hope. I wish I had the kind of time it takes to actually dive into these sorts of theoretical questions, but alas I do not, or at least I’ve made a triage decision not to.

To donate, click here. For larger amounts contact directly at vanessa@alter.org.il

Mathematical Metaphysics Institute

Focus: Searching for a mathematical basis for metaethics.

Leader: Alex Zhu

Funding Needed: Low

Confidence Level: Low

Alex Zhu has run iterations of the Math & Metaphysics Symposia, which had some excellent people in attendance, and intends partly to do more things of that nature. He thinks eastern philosophy contains much wisdom relevant to developing a future ‘decision-theoretic basis of metaethics’ and plans on an 8+ year project to do that.

I’ve seen plenty of signs that the whole thing is rather bonkers, but also strong endorsements from a bunch of people I trust that there is good stuff here, and the kind of crazy that is sometimes crazy enough to work. So there’s a lot of upside. If you think this kind of approach has a chance of working, this could be very exciting. For additional information, you can see this google doc.

To donate, message Alex at alex@mathematicalmetaphysics.org.

Focal at CMU

Focus: Game theory for cooperation by autonomous AI agents

Leader: Vincent Conitzer

Funding Needed: Medium

Confidence Level: Low

This is an area MIRI and the old rationalist crowd thought about a lot back in the day. There are a lot of ways for advanced intelligences to cooperate that are not available to humans, especially if they are capable of doing things in the class of sharing source code or can show their decisions are correlated with each other.

With sufficient capability, any group of agents should be able to act as if it is a single agent, and we shouldn’t need to do the game theory for them in advance either. I think it’s good things to be considering, but one should worry that even if they do find answers it will be ‘into the void’ and not accomplish anything. Based on my technical analysis I wasn’t convinced Focal was going to sufficiently interesting places with it, but I’m not at all confident in that assessment.

They note they’re also interested in the dynamics prior to Ai becoming superintelligent, as the initial conditions plausibly matter a lot.

To donate, reach out to Vincent directly at conitzer@cs.cmu.edu to be guided through the donation process.

Organizations Doing Cool Other Stuff Including Tech

This section is the most fun. You get unique projects taking big swings.

ALLFED

Focus: Feeding people with resilient foods after a potential nuclear war

Leaders: David Denkenberger

Funding Needed: High

Confidence Level: Medium

As far as I know, no one else is doing the work ALLFED is doing. A resilient food supply ready to go in the wake of a nuclear war (or other major disaster with similar dynamics) could be everything. There’s a small but real chance that the impact is enormous. In my 2021 SFF round, I went back and forth with them several times over various issues, ultimately funding them, you can read about those details here.

I think all of the concerns and unknowns from last time essentially still hold, as does the upside case, so it’s a question of prioritization, how likely you view nuclear war scenarios and how much promise you see in the tech.

If you are convinced by the viability of the tech and ability to execute, then there’s a strong case that this is a very good use of funds.

I think that this is a relatively better choice if you expect AI to remain a normal technology for a while or if your model of AI risks includes a large chance of leading to a nuclear war or other cascading impacts to human survival, versus if you don’t think this.

Research and investigation on the technical details seems valuable here. If we do have a viable path to alternative foods and don’t fund it, that’s a pretty large miss, and I find it highly plausible that this could be super doable and yet not otherwise done.

Donate here, or reach out to info@allfed.info.

Good Ancestor Foundation

Focus: Collaborations for tools to increase civilizational robustness to catastrophes

Leader: Colby Thompson

Funding Needed: High

Confident Level: High

The principle of ‘a little preparation now can make a huge difference to resilience and robustness in a disaster later, so it’s worth doing even if the disaster is not so likely’ generalizes. Thus, the Good Ancestor Foundation, targeting nuclear war, solar flares, internet and cyber outages, and some AI scenarios and safety work.

A particular focus is archiving data and tools, enhancing synchronization systems and designing a novel emergency satellite system (first one goes up in June) to help with coordination in the face of disasters. They’re also coordinating on hardening critical infrastructure and addressing geopolitical and human rights concerns.

They’ve also given out millions in regrants.

One way I know they make good decisions is they continue to help facilitate the funding for my work, and make that process easy. They have my sincerest thanks. Which also means there is a conflict of interest, so take that into account.

Donate here, or contact them at good@goodancestor.com.

Charter Cities Institute

Focus: Building charter cities

Leader: Kurtis Lockhart

Funding Needed: Medium

Confidence Level: Medium

I do love charter cities. There is little question they are attempting to do a very good thing and are sincerely going to attempt to build a charter city in Africa, where such things are badly needed. Very much another case of it being great that someone is attempting to do this so people can enjoy better institutions, even if it’s not the version of it I would prefer that would focus on regulatory arbitrage more.

Seems like a great place for people who don’t think transformational AI is on its way but do understand the value here.

Donate to them here, or contact them via webform.

Carbon Copies for Independent Minds

Focus: Whole brain emulation

Leader: Randal Koene

Funding Needed: Medium

Confidence Level: Low

At this point, if it worked in time to matter, I would be willing to roll the dice on emulations. What I don’t have is much belief that it will work, or the time to do a detailed investigation into the science. So flagging here, because if you look into the science and you think there is a decent chance, this becomes a good thing to fund.

Donate here, or contact them at contact@carboncopies.org.

Organizations Focused Primarily on Bio Risk Secure DNA

Focus: Scanning DNA synthesis for potential hazards

Leader: Kevin Esvelt, Andrew Yao and Raphael Egger

Funding Needed: Medium

Confidence Level: Medium

It is certainly an excellent idea. Give everyone fast, free, cryptographically screening of potential DNA synthesis to ensure no one is trying to create something we do not want anyone to create. AI only makes this concern more urgent. I didn’t have time to investigate and confirm this is the real deal as I had other priorities even if it was, but certainly someone should be doing this.

There is also another related effort, Secure Bio, if you want to go all out. I would fund Secure DNA first.

To donate, contact them at contact@securedna.org.

Blueprint Biosecurity

Focus: Increasing capability to respond to future pandemics, Next-gen PPE, Far-UVC.

Leader: Jake Swett

Funding Needed: Medium

Confidence Level: Medium

There is no question we should be spending vastly more on pandemic preparedness, including far more on developing and stockpiling superior PPE and in Far-UVC. It is rather a shameful that we are not doing that, and Blueprint Biosecurity plausibly can move substantial additional investment there. I’m definitely all for that.

To donate, reach out to donations@blueprintbiosecurity.org or head to the Blueprint Bio PayPal Giving Fund.

Pour Domain

Focus: EU policy for AI enabled biorisks, among other things.

Leader: Patrick Stadler

Funding Needed: Low

Confidence Level: Low

Everything individually looks worthwhile but also rather scattershot. Then again, who am I to complain about a campaign for e.g. improved air quality? My worry is still that this is a small operation trying to do far too much, some of it that I wouldn’t rank too high as a priority, and it needs more focus, on top of not having that clear big win yet. They are a French nonprofit.

Donation details are at the very bottom of this page, or you can contact them at info@pourdemain.ngo.

ALTER Israel

Focus: AI safety and biorisk for Israel

Leader: David Manheim

Funding Needed: Low

Confidence Level: Medium

Israel has Ilya’s company SSI (Safe Superintelligence) and otherwise often punches above its weight in such matters but is getting little attention. This isn’t where my attention is focused but David is presumably choosing this focus for good reason.

To support them, get in touch at contact@alter.org.il.

Organizations That Can Advise You Further

The first best solution, as I note above, is to do your own research, form your own priorities and make your own decisions. This is especially true if you can find otherwise illegible or hard-to-fund prospects.

However, your time is valuable and limited, and others can be in better positions to advise you on key information and find opportunities.

Another approach to this problem, if you have limited time or actively want to not be in control of these decisions, is to give to regranting organizations, and take the decisions further out of your own hands.

Effective Institutions Project (EIP) (As A Donation Advisor)

Focus: AI governance, advisory and research, finding how to change decision points

Leader: Ian David Moss

Confidence Level: High

I discussed their direct initiatives earlier. This is listing them as a donation advisor and in their capacity of attempting to be a resource to the broader philanthropic community.

They report that they are advising multiple major donors, and would welcome the opportunity to advise additional major donors. I haven’t had the opportunity to review their donation advisory work, but what I have seen in other areas gives me confidence. They specialize in advising donors who have brad interests across multiple areas, and they list AI safety, global health, democracy and (peace and security).

To donate, click here. If you have further questions or would like to be advised, contact them at info@effectiveinstitutionsproject.org.

Longview Philanthropy

Focus: Conferences and advice on x-risk for those giving >$1 million per year

Leader: Simran Dhaliwal

Funding Needed: None

Confidence Level: Low

Longview is not seeking funding, instead they are offering support to large donors, and you can give to their regranting funds, including the Emerging Challenges Fund on catastrophic risks from emerging tech, which focuses non-exclusively on AI.

I had a chance to hear a pitch for them at The Curve and check out their current analysis and donation portfolio. It was a good discussion. There were definitely some areas of disagreement in both decisions and overall philosophy, and I worry they’ll be too drawn to the central and legible (a common issue with such services).

On the plus side, they’re clearly trying, and their portfolio definitely had some good things in it. So I wouldn’t want to depend on them or use them as a sole source if I had the opportunity to do something higher effort, but if I was donating on my own I’d find their analysis useful. If you’re considering relying heavily on them or donating to the funds, I’d look at the fund portfolios in detail and see what you think.

I pointed them to some organizations they hadn’t had a chance to evaluate yet.

They clearly seem open to donations aimed at particular RFPs or goals.

To inquire about their services, contact them at info@longview.org.

Organizations That then Regrant to Fund Other Organizations

There were lots of great opportunities in SFF in both of my recent rounds. I was going to have an embarrassment of riches I was excited to fund.

Thus I decided quickly that I would not be funding any regrating organizations. If you were in the business of taking in money and then shipping it out to worthy causes, well, I could ship directly to highly worthy causes.

So there was no need to have someone else do that, or expect them to do better.

That does not mean that others should not consider such donations.

I see three important advantages to this path.

Regranters can offer smaller grants that are well-targeted.
Regranters save you a lot of time.
Regranters avoid having others try to pitch on donations.

Thus, if you are making a ‘low effort’ donation, and think others you trust that share your values to invest more effort, it makes more sense to consider regranters.

In particular, if you’re looking to go large, I’ve been impressed by SFF itself, and there’s room for SFF to scale both its amounts distributed and level of rigor.

SFF Itself (!)

Focus: Give out grants based on recommenders, primarily to 501c(3) organizations

Leaders: Andrew Critch and Jaan Tallinn

Funding Needed: High

Confidence Level: High

If I had to choose a regranter right now to get a large amount of funding, my pick would be to partner with and participate in the SFF process as an additional funder. The applicants and recommenders are already putting in their effort, with plenty of room for each round to scale. It is very clear there are plenty of exciting places to put additional funds.

With more funding, the decisions could improve further, as recommenders would be better motivated to devote more time, and we could use a small portion of additional funds to make them better resourced.

The downside is that SFF can’t ‘go small’ efficiently on either funders or causes.

SFF does not accept donations but they are interested in partnerships with people or institutions who are interested in participating as a Funder in a future S-Process round. The minimum requirement for contributing as a Funder to a round is $250k. They are particularly interested in forming partnerships with American donors to help address funding gaps in 501(c)(4)’s and other political organizations.

This is a good choice if you’re looking to go large and not looking to ultimately funnel towards relatively small funding opportunities or individuals.

Manifund

Focus: Regranters to AI safety, existential risk, EA meta projects, creative mechanisms

Leader: Austin Chen (austin at manifund.org).

Funding Needed: Medium

Confidence Level: Medium

This is a regranter that gives its money to its own regranters, one of which was me, for unrestricted grants. They’re the charity donation offshoot of Manifold. They’ve played with crowdfunding, and with impact certificates, and ACX grants. They help run Manifest.

You’re essentially hiring these people to keep building a website and trying alternative funding allocation mechanisms, and for them to trust the judgment of selected regranters. That seems like a reasonable thing to do if you don’t otherwise know where to put your funds and want to fall back on a wisdom of crowds of sorts. Or, perhaps, if you actively want to fund the cool website.

Manifold itself did not apply, but I would think that would also be a good place to invest or donate in order to improve the world. It wouldn’t even be crazy to go around subsidizing various markets. If you send me manna there, I will set aside and use that manna to subsidize markets when it seems like the place to do that.

If you want to support Manifold itself, you can either donate or buy a SAFE by contacting Austin at austin@manifund.org.

Also I’m a regranter at Manifund, so if you wanted to, you could use that to entrust me with funds to regrant. As you can see I certainly feel I have plenty of good options here if I can’t find a better local one, and if it’s a substantial amount I’m open to general directions (e.g. ensuring it happens relatively quickly, or a particular cause area as long as I think it’s net positive, or the method of action or theory of impact). However, I’m swamped for time, so I’d probably rely mostly on what I already know.

AI Risk Mitigation Fund

Focus: Spinoff of LTFF, grants for AI safety projects

Leader: Thomas Larsen

Funding Needed: Medium

Confidence Level: High

Seems very straightforwardly exactly what it is, a regranter that is usually in the low six figure range. Fellow recommenders were high on Larsen’s ability to judge projects. If you think this is better than you can do on your own and you want to fund such projects, then go for it.

I’ve talked to them on background about their future plans and directions, and without sharing details their plans make me more excited here.

Donate here or contact them at info@airiskfund.com.

Long Term Future Fund

Focus: Grants of 4-6 figures mostly to individuals, mostly for AI existential risk

Leader: Caleb Parikh (among other fund managers)

Funding Needed: High

Confidence Level: Low

The pitch on LTFF is that it is a place for existential risk people who need modest cash infusions to ask for them, and to get them without too much overhead or distortion. Looking over the list of grants, there is at least a decent hit rate.

One question is, are the marginal grants a lot less effective than the average grant?

My worry is that I don’t know the extent to which the process is accurate, fair, favors insiders or extracts a time or psychic tax on participants, favors legibility, or rewards ‘being in the EA ecosystem’ or especially the extent to which the net effects are distortionary and bias towards legibility and standardized efforts. Or the extent to which people use the system to extract funds without actually doing anything.

That’s not a ‘I think this is bad,’ it is a true ‘I do not know.’ I doubt they know either.

What do we know? They say applications should take 1-2 hours to write and between 10 minutes and 10 hours to evaluate, although that does not include time forming the plan, and this is anticipated to be an ~yearly process long term. And I don’t love that this concern is not listed under reasons not to choose to donate to the fund (although the existence of that list at all is most welcome, and the reasons to donate don’t consider the flip side either).

Given their current relationship to EA funds, you likely should consider LTFF if and only if you both want to focus on AI existential risk via regrants and also want to empower and strengthen the existing EA formal structures and general ways of being.

That’s not my preference, but it could be yours.

Donate here, or contact the fund managers at longtermfuture@effectivealtruismfunds.org.

Foresight

Focus: Regrants, fellowships and events

Leader: Allison Duettmann

Funding Needed: Medium

Confidence Level: Low

Foresight also does other things. I’m focusing here on their AI existential risk grants, which they offer on a rolling basis. I’ve advised them on a small number of potential grants, but they rarely ask.

The advantage on the regrant side would be to get outreach that wasn’t locked too tightly into the standard ecosystem. The other Foresight activities all seem clearly like good things, but the bar these days is high and since they weren’t the topic of the application I didn’t investigate.

Donate here, or reach out to foresight@foresight.org.

Centre for Enabling Effective Altruism Learning & Research (CEELAR)

Focus: Strategic incubator and launchpad for EA talent, research, and high-impact initiatives, with emphasis on AI safety, GCR reduction, and longtermist work

Leader: Attila Ujvari

Funding Needed: High

Confidence Level: Low

I loved the simple core concept of a ‘catered hotel’ where select people can go to be supported in whatever efforts seem worthwhile. They are now broadening their approach, scaling up and focusing on logistical and community supports, incubation and a general infrastructure play on top of their hotel. This feels less unique to me now and more of a typical (EA UK) community play, so you should evaluate it on that basis.

Donate here, or reach out to contact@ceealar.org.

Organizations That are Essentially Talent Funnels

I am less skeptical of prioritizing AI safety talent funnels than I was last year, but I remain skeptical.

The central reason remains simple. If we have so many good organizations already, in need of so much funding, why do we need more talent funnels? Is talent our limiting factor? Are we actually in danger of losing important talent?

The clear exception is leadership and management. There remains, it appears, a clear shortage of leadership and management talent across all charitable space, and startup space, and probably flat out all of space.

Which means if you are considering stepping up and doing leadership and management, then that is likely more impactful than you might at first think.

If there was a strong talent funnel specifically for leadership or management, that would be a very interesting funding opportunity. And yes, of course there still need to be some talent funnels. Right now, my guess is we have enough, and marginal effort is best spent elsewhere.

What about for other talent? What about placements in government, or in the AI labs especially Anthropic of people dedicated to safety? What about the prospects for much higher funding availability by the time we are ready to put people to work?

If you can pull it off, empowering talent can have a large force multiplier, and the opportunity space looks better than a year ago. It seems plausible that frontier labs will soak up every strong safety candidate they can find, since the marginal returns there are very high and needs are growing rapidly.

Secondary worries include the danger you end up feeding capability researchers to AI labs, and the discount for the time delays involved.

My hunch is this will still receive relatively more attention and funding than is optimal, but marginal funds here will still be useful if deployed in places that are careful to avoid being lab talent funnels.

AI Safety Camp

Focus: Learning by doing, participants work on a concrete project in the field

Leaders: Remmelt Ellen and Linda Linsefors and Robert Kralisch

Funding Needed: Low

Confidence Level: High

By all accounts they are the gold standard for this type of thing. Everyone says they are great, I am generally a fan of the format, I buy that this can punch way above its weight or cost. If I was going to back something in this section, I’d start here.

Donors can reach out to Remmelt at remmelt@aisafety.camp, or leave a matched donation to support next projects.

Center for Law and AI Risk

Focus: Paying academics small stipends to move into AI safety work

Leaders: Peter Salib (psalib @ central.uh.edu), Yonathan Arbel (yarbel @ law.ua.edu) and Kevin Frazier (kevin.frazier @ law.utexas.edu).

Funding Needed: Low

Confidence Level: High

This strategy is potentially super efficient. You have an academic that is mostly funded anyway, and they respond to remarkably small incentives to do something they are already curious about doing. Then maybe they keep going, again with academic funding. If you’re going to do ‘field building’ and talent funnel in a world short on funds for those people, this is doubly efficient. I like it. They’re now moving into hiring an academic fellow, the theory being ~1 year of support to create a permanent new AI safety law professor.

To donate, message one of leaders at the emails listed above.

Speculative Technologies

Focus: Enabling ambitious research programs that are poor fits for both academia and VC-funded startups including but not limited to Drexlerian functional nanomachines, high-throughput tools and discovering new superconductors.

Leader: Benjamin Reinhardt

Funding Needed: Medium

Confidence Level: Medium

I have confirmation that Reinhardt knows his stuff, and we certainly could use more people attempting to build revolutionary hardware. If the AI is scary enough to make you not want to build the hardware, it would figure out how to build the hardware anyway. You might as well find out now.

If you’re looking to fund a talent funnel, this seems like a good choice.

To donate, go here or reach out to info@spec.tech.

Talos Network

Focus: Fellowships to other organizations, such as Future Society, Safer AI and FLI.

Leader: Chiara Gerosa

Funding Needed: Medium

Confidence Level: Low

They run two fellowship cohorts a year. They seem to place people into a variety of solid organizations, and are exploring the ability to get people into various international organizations like the OECD, UN or European Commission or EU AI Office.

The more I am convinced people will actually get inside meaningful government posts, the more excited I will be.

To donate, contact team@talosnetwork.org.

MATS Research

Focus: Researcher mentorship for those new to AI safety.

Leaders: Ryan Kidd and Christian Smith.

Funding Needed: High

Confidence Level: Medium

MATS is by all accounts very good at what they do and they have good positive spillover effects on the surrounding ecosystem. The recruiting classes they’re getting are outstanding.

If (and only if) you think that what they do, which is support would-be alignment researchers starting out and especially transitioning from other professions, is what you want to fund, then you should absolutely fund them. That’s a question of prioritization.

Donate here, or contact them via webform.

Epistea

Focus: X-risk residencies, workshops, coworking in Prague, fiscal sponsorships

Leader: Irena Kotikova

Funding Needed: Medium

Confidence Level: Medium

I see essentially two distinct things here.

First, you have the umbrella organization, offering fiscal sponsorship for other organizations. Based on what I know from the charity space, this is a highly valuable service – it was very annoying getting Balsa a fiscal sponsor while we waited to become a full 501c3, even though we ultimately found a very good one that did us a solid, and also annoying figuring out how to be on our own going forward.

Second, you have various projects around Prague, which seem like solid offerings in that class of action of building up EA-style x-risk actions in the area, if that is what you are looking for. So you’d be supporting some mix of those two things.

To donate, contact hello@epistea.org.

Emergent Ventures

Focus: Small grants to individuals to help them develop their talent

Leader: Tyler Cowen

Funding Needed: Medium

Confidence Level: High

Emergent Ventures are not like the other talent funnels in several important ways.

It’s not about AI Safety. You can definitely apply for an AI Safety purpose, he’s granted such applications in the past, but it’s rare and topics run across the board, well beyond the range otherwise described in this post.
Decisions are quick and don’t require paperwork or looking legible. Tyler Cowen makes the decision, and there’s no reason to spend much time on your end either.
There isn’t a particular cause area this is trying to advance. He’s not trying to steer people to do any particular thing. Just to be more ambitious, and be able to get off the ground and build connections and so on. It’s not prescriptive.

I strongly believe this is an excellent way to boost the development of more talent, as long as money is serving as a limiting factor on the project, and that it is great to develop talent even if you don’t get to direct or know where it is heading. Sure, I get into rhetorical arguments with Tyler Cowen all the time, around AI and also other things, and we disagree strongly about some of the most important questions where I don’t understand how he can continue to have the views he expresses, but this here is still a great project, an amazingly cost-efficient intervention.

Donate here (specify “Emergent Ventures” in notes), or reach out to emergentventures@mercatus.gmu.edu.

AI Safety Cape Town

Focus: AI safety community building and research in South Africa

Leaders: Leo Hyams and Benjamin Sturgeon

Funding Needed: Low

Confidence Level: Low

This is a mix of AI research and building up the local AI safety community. One person whose opinion I value gave the plan and those involved in it a strong endorsement, so including it based on that.

To donate, reach out to leo@aisafetyct.com.

ILINA Program

Focus: Talent for AI safety in Africa

Leaders: Cecil Abungu

Funding Needed: Low

Confidence Level: Low

I have a strong endorsement in hand in terms of their past work, if you think this is a good place to go in search of talent.

To donate, reach out to cecil@ilinaprogram.org.

Impact Academy Limited

Focus: Global talent accelerator and hiring partner for technical AI safety, supporting worker transitions into AI safety.

Leader: Roy Hagemann and Varun Agarwal

Funding Needed: Medium

Confidence Level: Low

They previously focused on India, one place with lots of talent, they’re now global. A lot has turned over in the last year, so you’ll want to check them out anew.

To donate, contact info@impactacademy.org.

Atlas Computing

Focus: Mapping & creating missing orgs for AI safety (aka Charity Entrepreneurship for AI risk)

Leaders: Evan Miyazono

Funding Needed: Medium

Confidence Level: Low

There was a pivot this past year from technical research to creating ‘missing orgs’ in the AI risk space. That makes sense as a strategy if and only if you expect the funding necessary to come in, or you think they can do especially strong targeting. Given the change they will need to be reevaluated.

They receive donations from here, or you can email them at hello@atlascomputing.org.

Principles of Intelligence (Formerly PIBBSS)

Focus: Fellowships and affiliate programs for new alignment researchers

Leader: Lucas Teixeira and Dusan D. Nesic

Funding Needed: High

Confidence Level: Low

There are some hits here. Gabriel Weil in particular has impressed me in our interactions and with his work and they cite a good technical paper. But also that’s with a lot of shots on goal, and I’d have liked to see some bigger hits by now.

A breakdown revealed that, largely because they start with relatively senior people, most of them get placed in a way that doesn’t require additional support. That makes them a better bet than many similar rivals.

To donate, reach out to contact@princint.ai, or fund them through Manifund here.

Tarbell Center

Focus: Journalism fellowships for oversight of AI companies.

Leader: Cillian Crosson (Ex-Talos Network; still on their board.)

Funding Needed: High

Confidence Level: Medium

They offer fellowships to support journalism that helps society navigate the emergence of increasingly advanced AI, and a few other journalism ventures. They have sponsored at least one person who went on to do good work in the area. They also sponsor article placement, which seems reasonably priced in the grand scheme of things, I think?

I am not sure this is a place we need to do more investment, or if people trying to do this even need fellowships. Hard to say. There’s certainly a lot more tech reporting and more every day, if I’m ever short of material I have no trouble finding more.

It is still a small amount of money per person that can meaningfully help people get on their feet and do something useful. We do in general need better journalism. They seem to be in a solid place but also I’d be fine with giving a bunch more funding to play with, they seem pretty unique.

Donate here, or reach out to them via webform.

Catalyze Impact

Focus: Incubation of AI safety organizations

Leader: Alexandra Bos

Funding Needed: Medium

Confidence Level: Low

Why funnel individual talent when you can incubate entire organizations? I am not convinced that on the margin we currently need more of either, but I’m more receptive to the idea of an incubator. Certainly incubators can be high leverage points for getting valuable new orgs and companies off the ground, especially if your model is that once the org becomes fundable it can unlock additional funding.

If you think an incubator is worth funding, then the question is whether this is the right team. The application was solid all around, and their track record includes Timaeus and Carma, although counterfactuals are always difficult. Beyond that I don’t have a differentiator on why this is the team.

To donate, contact them at info@catalyze-impact.org.

CeSIA within EffiSciences

Focus: New AI safety org in Paris, discourse, R&D collaborations, talent pipeline

Leaders: Charbel-Raphael Segerie, Florent Berthet

Funding Needed: Low

Confidence Level: Low

They’re doing all three of discourse, direct work and talent funnels. They run the only university AI safety course in Europe, maintain the AI Safety Atlas, and have had their recommendations integrated verbatim into the EU AI Act’s Code of Practice. Their two main priorities are supporting the enforcement of the EU AI Act, and driving international agreements on AI red lines.

To donate, go here, or contact them at contact@securite-ia.fr.

Stanford Existential Risk Initiative (SERI)

Focus: Recruitment for existential risk causes

Leader: Steve Luby

Funding Needed: Medium

Confidence Level: Low

Stanford students certainly are one place to find people worth educating about existential risk. It’s also an expensive place to be doing it, and a place that shouldn’t need extra funding. And that hates fun. And it’s not great that AI is listed third on their existential risk definition. So I’m not high on them, but it sure beats giving unrestricted funds to your Alma Mater.

Interested donors should contact Steve Luby directly at sluby@stanford.edu.

Non-Trivial

Focus: Talent funnel directly to AI safety and biosecurity out of high school

Leader: Peter McIntyre

Funding Needed: Low

Confidence Level: Low

Having high school students jump straight to research and placement sounds good to me, and plausibly the best version of a talent funnel investment. I haven’t confirmed details but I like the theory.

To donate, get in touch at info@non-trivial.org.

CFAR

Focus: Teaching rationality skills, seeking to make sense of the world and how to think

Leader: Anna Salamon

Funding Needed: High

Confidence Level: High

I am on the board of CFAR, so there is a direct and obvious conflict. Of course, I am on the board of CFAR exactly because I think this is a worthwhile use of my time, and also because Anna asked me. I’ve been involved in various ways since the beginning, including the discussions about whether and how to create CFAR in the first place.

CFAR is undergoing an attempted revival. There weren’t workshops for many years, for a variety of reasons including safety concerns and also a need to reorient. The workshops are now starting up again, with a mix of both old and new units, and I find much of the new material interesting and potentially valuable. I’d encourage people to consider attending workshops, and also donating.

To donate, click here, or reach out to contact@rationality.org.

The Bramble Center

Focus: Workshops in the style of CFAR but focused on practical courage, forming high value relationships between attendees with different skill sets and learning to care for lineages, in the hopes of repairing the anglosphere and creating new capable people to solve our problems including AI in more grounded ways.

Leader: Anna Salamon

Funding Needed: Low

Confidence Level: High

LARC is kind of a spin-off of CFAR, a place to pursue a different kind of agenda. I absolutely do not have high confidence that this will succeed, but I do have high confidence that this is a gamble worth taking, and that if those involved here (especially Anna Salamon but also others that I know) want to devote their time to trying this, that we should absolutely give them that opportunity.

Donate here.

Final Reminders

If an organization was not included here, or was removed for the 2025 edition, again, that does not mean they aren’t good, or even that I wouldn’t endorse them if asked.

It could be because I am not aware of the organization, or lack sufficient knowledge at this point to be confident in listing them, or I fear my knowledge is obsolete.

It could be that they asked to be excluded, which happened in several cases.

If by accident I included you and you didn’t want to be included and I failed to remove you, or you don’t like the quote here, I sincerely apologize and will edit you out right away, no questions asked.

If an organization is included here, that is a good thing, but again, it does not mean you should donate without checking if it makes sense based on what you think is true, how you think the world works, what you value and what your priorities are. There are no universal right answers.

Discuss

How Reducing Cognitive Interference Could Revolutionize Stroke Recovery

27 ноября, 2025 - 18:34

Published on November 27, 2025 3:34 PM GMT

The most frustrating truth in motor learning is that thinking too hard about how you move can make you worse. Not "trying hard" in the sense of effort or volume, but "trying hard" in the sense of conscious, deliberate attention to mechanics. When a pianist thinks too carefully about which fingers to move, their playing becomes stilted. When a basketball player consciously analyzes their shooting form mid-shot, they miss. When a stroke patient over-focuses on the individual components of reaching for a cup, their recovery slows.

This phenomenon has a name: the reinvestment hypothesis, and it suggests that explicit, conscious monitoring of motor tasks actively interferes with the implicit, procedural systems that actually execute skilled movement. For stroke survivors trying to regain the ability to move a paralyzed arm, this creates a devastating paradox. They're desperate to move, they focus intensely on moving, and that very focus may be sabotaging their recovery.

What if we could temporarily turn off that interference? What if we could create a neural state where the overthinking stops, where the brain's executive control systems step aside and let the motor learning circuits do their work unimpeded? And what if we could do this precisely during the moments when the brain is receiving the most valuable training signal, when motor intent is perfectly paired with sensory feedback?

This isn't science fiction. The pieces already exist. What's needed is putting them together in the right way.

Why Stroke Recovery Takes So Long

When someone has a stroke affecting their motor cortex or corticospinal tract, the fundamental problem is simple, the brain's signals can no longer reach certain muscles. The neural highway is damaged. What remains is a patient who can imagine moving their arm, who desperately wants to move their arm, but whose arm refuses to cooperate.

Traditional rehabilitation tries to work around this through sheer repetition. Physical therapists guide patients through thousands of movement attempts, hoping that spared pathways will gradually take over the functions of the damaged ones. Modern approaches add technology: robotic exoskeletons that assist movement, high-intensity task-specific training programs, even virtual reality environments. The evidence base for these interventions is solid. Constraint-Induced Movement Therapy shows real improvements in upper-limb outcomes, action observation combined with motor imagery produces meaningful gains, and high-dose practice matters.

But even with all these advances, stroke recovery remains frustratingly slow. A patient might spend six months of intensive therapy just to regain the ability to grasp a cup. During that time, they can't work, can't care for themselves fully, can't return to the life they had. The functional improvements plateau. The neural reorganization hits limits.

The question is: why?

Part of the answer comes from an unexpected place: research on how people learn implicit versus explicit motor skills. It turns out that when stroke patients are given explicit instructions about how to perform a movement, "extend your wrist like this," "focus on your elbow angle," they often learn more slowly than when they're allowed to discover the movement through practice alone. The conscious, effortful attention to movement mechanics that seems like it should help is actually getting in the way.

This makes sense from a neural architecture perspective. Motor learning relies on cortico-basal ganglia circuits for sequence chunking and action selection, and cortico-cerebellar circuits for error-based calibration and timing. These are largely implicit, procedural systems. Meanwhile, explicit motor control recruits prefrontal and parietal areas, the same systems involved in working memory, executive function, and conscious attention. When stroke patients consciously micromanage their movements, they're essentially forcing a top-down, executive-driven process onto a system that works better through bottom-up, automatic learning.

This pattern holds up in the lab. When people are instructed to focus on the effects of their movement ("move the cup to the target") rather than their body parts ("extend your arm"), they learn faster and perform better. The mechanism appears to be that external focus allows the motor system to self-organize without interference from conscious monitoring. Though the effect size estimates have been debated, the direction is consistent: less conscious control often means better motor learning.

This creates a problem for stroke patients. They have every reason to consciously focus on their movements, those movements don't work, they're effortful, they require intense concentration just to produce any result at all. The very difficulty of the task seems to demand explicit attention. But that attention may be part of what's keeping them stuck.

What's needed is a way to short-circuit this pattern. To provide the brain with successful movement experiences while simultaneously reducing the executive interference that slows procedural learning. To create conditions where implicit, automatic motor learning can flourish.

Closing the Loop with BCI-FES

Before we can address the cognitive interference problem, we need to solve the more fundamental issue: how do you give a stroke patient successful movement experiences when their brain signals can't reach their muscles?

The answer comes from combining two technologies: brain-computer interfaces (BCIs) and functional electrical stimulation (FES). A BCI reads electrical activity from the brain, typically using an EEG to detect motor-related patterns, and translates it into a control signal. When a patient imagines or attempts to move their paralyzed hand, the BCI detects the motor imagery signature (a decrease in mu and beta frequency power over the motor cortex, called event-related desynchronization or ERD) and uses it to trigger an external device.

FES, meanwhile, delivers small electrical currents directly to muscles, causing them to contract. The stimulation bypasses the damaged neural pathways entirely, creating movement through peripheral activation.

Put them together and you get something remarkable: a closed-loop system where motor intent directly causes motor action, even when the natural pathway is broken. The patient thinks about moving their hand, the BCI detects this intention, and FES makes their hand actually move. The brain's command reaches the muscle, just through an artificial route.

Why does this matter? Because of Hebbian plasticity, the principle that neurons that fire together wire together. When the brain generates a motor command and immediately receives sensory feedback that the movement actually happened, it strengthens the entire sensorimotor loop. The intent and the consequence are temporally linked, creating an associative learning signal. Over time, this should help rebuild functional neural pathways, allowing the patient to eventually move without needing the assistive technology.

Figure 1: Effect of BCI-Based Training on FMA-UE by Feedback Type

Mean difference in Fugl-Meyer Upper Extremity scores and 95% confidence intervals from Li et al., JNER 2025 meta-analysis of 21 RCTs (n=886). BCI+FES shows the strongest effect (MD = 4.37 points), outperforming BCI with robotic or visual feedback alone.

And it works. The evidence is now substantial. A 2025 meta-analysis of 21 randomized controlled trials including 886 patients found that BCI-based training produces clinically meaningful improvements in upper-limb function, with an overall mean difference of 3.69 points on the Fugl-Meyer Assessment Upper Extremity scale (FMA-UE). The BCI+FES subgroup performed best, with a mean difference of 4.37 points, well above the minimal clinically important difference of 4-5 points.

These effects appear in both subacute stroke (mean difference 5.31 points, where plasticity is still high) and chronic stroke (mean difference 2.63-3.71 points, where spontaneous recovery has plateaued). The benefits extend beyond the immediately trained movements, with improvements seen on the Wolf Motor Function Test and Action Research Arm Test as well. Most importantly, the gains are durable, multiple studies show maintained improvements at 6-12 month follow-up.

The mechanism appears to be genuine neural reorganization, not just temporary facilitation. EEG and fMRI studies show that patients undergoing BCI-FES training develop increased connectivity in motor areas of the affected hemisphere, with the degree of connectivity change correlating with functional improvement. The brain is actually rewiring.

Perhaps most telling, BCI-FES outperforms both components used alone. BCI with visual feedback is better than traditional therapy, but BCI with actual movement (FES) is better still. The closed loop matters. The temporal pairing of intent and sensory consequence matters. In a double-blind RCT, patients receiving motor-imagery-contingent FES (where the stimulation only occurred when they successfully generated the right brain pattern) showed greater improvements than patients receiving the same amount of FES on a fixed schedule. The contingency, the fact that their intention caused the movement, was therapeutically important.

This makes BCI-FES a strong foundation for rehabilitation. But there's still room for improvement. The effect sizes, while clinically meaningful, are modest. The between-patient variability is substantial. Some patients show dramatic recovery; others show minimal change. And the question remains: if we're successfully creating the conditions for Hebbian learning through intent-contingent feedback, why aren't the effects even larger?

One possibility is that we're fighting the executive interference problem. Yes, the BCI-FES provides the motor signal, but patients are still consciously trying, monitoring, and analyzing their attempts. Their prefrontal cortex is still in the way.

What Happens When Top-Down Control Is Reduced

Here's where things get interesting. There's a rare neurological phenomenon called savant syndrome, where individuals with significant developmental or cognitive disabilities exhibit "islands of genius," isolated areas of exceptional ability that contrast sharply with their overall impairment. People with savant syndrome can perform lightning-fast calendar calculations, reproduce complex musical pieces after a single hearing, create photorealistic drawings from memory.

The traditional explanation focused on enhanced local processing or specialized neural architecture. But starting in the 2000s, neuroscientist Allan Snyder proposed something more provocative: what if savant abilities aren't special skills that most people lack, but rather latent capabilities that most brains suppress? What if the savant's abilities emerge not from having something extra, but from lacking certain top-down inhibitory processes that normally filter and constrain cognition?

The hypothesis is that typical brains engage in constant top-down control, applying rules, strategies, learned patterns, and executive oversight to sensory and motor processing. This is usually adaptive; it lets us categorize, generalize, and operate efficiently. But it also means we can't access the raw, unfiltered information that our lower-level systems actually process. A savant with reduced prefrontal inhibition might "see" the individual features of a scene with unusual clarity because they're not being automatically chunked into higher-level categories.

The truly striking evidence came when researchers tried to temporarily create "savant-like" states in neurotypical people. In a series of controversial but replicated experiments, low-frequency repetitive transcranial magnetic stimulation (rTMS) applied to the left anterior temporal lobe - an area involved in top-down semantic and conceptual control - transiently improved performance on drawing tasks, proofreading, and even numerical estimation. The effect was small and temporary, but it suggested the principle was real: reducing certain types of top-down control can unmask capabilities that are normally suppressed.

Now, we're not trying to create savant abilities in stroke patients. They don't need to do calendar calculations or draw photorealistic portraits. But the underlying principle is directly applicable: what if the same executive systems that normally constrain cognition are also interfering with motor relearning?

We already know that explicit monitoring hurts motor learning. We know that prefrontal areas contribute to explicit, rule-based motor control. We know that stroke patients tend to over-engage these systems, consciously trying to control movements that should be automatic. What if we could temporarily reduce this prefrontal interference, creating a state where implicit motor learning could proceed more efficiently?

This is where the specific anatomical target matters. The left dorsolateral prefrontal cortex (left DLPFC, Brodmann areas 9 and 46) is heavily involved in working memory, executive control, and explicit strategy use. When you consciously think about how to perform a movement, when you monitor your performance and try to adjust it analytically, when you apply verbal instructions to motor control - that's largely left DLPFC activity.

And there's direct evidence that inhibiting left DLPFC can facilitate implicit learning. In a striking 2017 study, researchers applied continuous theta burst stimulation (cTBS), a rapid, inhibitory form of rTMS, to the left DLPFC while adults learned novel word forms. The cTBS group learned faster and showed better retention than both the sham stimulation group and a group receiving stimulation to the right DLPFC. The interpretation: by temporarily reducing the left DLPFC's top-down control, the inhibitory stimulation allowed implicit, statistical learning mechanisms to work more efficiently.

The same principle should apply to motor learning. In fact, it may apply even more strongly, since motor learning is fundamentally an implicit process. By using cTBS to transiently reduce left DLPFC activity during motor training, we might be able to shift patients from an explicit, effortful, self-monitoring mode into a more implicit, automatic, procedural learning mode. We'd be creating a temporary "flow state" where the overthinking stops and the motor system can do its job.

The Proposal: Strategic Neural Inhibition Paired with Closed-Loop Training

Here's the complete picture: take the proven BCI-FES approach that successfully closes the sensorimotor loop, and add a brief pre-session intervention that temporarily reduces executive interference. Specifically, apply 40 seconds of cTBS to the left DLPFC immediately before each BCI-FES training session.

The timing is critical. cTBS produces a reduction in cortical excitability that lasts approximately 30-60 minutes. This matches the typical length of a BCI-FES training block. The patient receives the cTBS, has a few minutes of setup time (EEG cap placement, FES electrode positioning, system calibration), and then begins training while their DLPFC is still inhibited. They're primed for implicit learning right when they're receiving the most valuable learning signal: the tight temporal pairing of motor intent and sensory feedback.

The technical parameters matter. For cTBS, the established protocol is 600 pulses delivered as 50 Hz triplets repeating at 5 Hz, taking exactly 40 seconds. The stimulation intensity is typically 70-80% of active motor threshold or a guideline-concordant percentage of resting motor threshold. The target is left DLPFC, which can be localized using individual structural MRI and neuronavigation for precision, or approximated using the scalp-based F3 position if imaging isn't available. These parameters come from Huang et al.'s seminal 2005 paper establishing the theta burst stimulation protocol, and they're explicitly covered in the 2021 International Federation of Clinical Neurophysiology safety guidelines.

For the BCI-FES component, the system uses standard EEG (16-32 channels, focused on sensorimotor areas C3/C4) to detect motor imagery or movement attempt through mu and beta band event-related desynchronization. Common spatial patterns (CSP) and linear discriminant analysis (LDA) provide the classification, though other machine learning approaches work as well. The key is the contingency: FES is only delivered when the patient successfully generates the appropriate brain pattern. This is calibrated each session, as brain signals change over time.

The FES parameters also matter. A common starting point is 50 Hz stimulation with a 300-microsecond pulse width, based on the parameters used in successful RCTs. The current amplitude is titrated to produce visible muscle contraction without discomfort, typically starting low and gradually increasing as the patient adapts. Early sessions might focus on wrist extension, but the approach can be extended to multi-channel stimulation patterns that produce more complex, functional movements as the patient improves.

Figure 2: Conceptual Schematic of the Intervention

The workflow is straightforward. The patient arrives, and a trained technician applies the cTBS using a TMS coil positioned over left DLPFC. This takes less than a minute once the target is located. After the 40-second stimulation burst, there's a brief transition period while the EEG cap and FES electrodes are placed and the system is calibrated (3-5 minutes). Then the patient begins the motor training: they repeatedly attempt or imagine moving their affected limb, the BCI detects successful attempts, and FES causes the actual movement. This continues for 40-60 minutes, during which time the DLPFC remains inhibited from the earlier cTBS.

Importantly, during the training period, explicit strategy coaching is minimized. The patient isn't given detailed instructions about how to move or what to think about. Instead, the focus is on external goals ("reach for the target") and letting the system provide feedback through successful movements. The reduced DLPFC activity should make this implicit approach more natural - with less executive monitoring, patients may find it easier to stop overthinking and just let the movements emerge.

What should this achieve? The mechanisms are complementary. The cTBS provides cognitive priming, setting the brain state to favor implicit learning. The BCI-FES provides contingent feedback, ensuring that motor intent is reliably paired with sensory consequences, creating strong Hebbian learning signals. Together, they address both the "what to learn" (the motor patterns that produce successful reaching, grasping, manipulation) and the "how to learn it" (through implicit, procedural mechanisms rather than explicit, executive-driven ones).

This is not two competing interventions but complementary components working at different levels. FES provides peripheral afference (sensory input from the stimulated muscles) that's temporally locked to central intent (the motor imagery that triggered the BCI). cTBS provides supramodal cognitive state optimization, reducing the top-down interference that would normally slow procedural learning. One ensures the motor system gets the right training signal; the other ensures the brain is in the right state to learn from it.

Expected Outcomes and Clinical Impact

Given that BCI-FES alone produces mean improvements of approximately 4.4 points on the FMA-UE scale, what additional benefit should we expect from adding cTBS-mediated cognitive priming?

The honest answer is, we don't know exactly, because this specific combination hasn't been tested. But we can make educated estimates based on the magnitude of effects seen when implicit learning conditions are optimized. Suppose the cognitive interference hypothesis is correct, and a substantial portion of the between-patient variability in BCI-FES outcomes reflects differences in how much executive monitoring interferes with learning. In that case, we might expect gains of 2-3 additional FMA-UE points above the BCI-FES baseline.

This may not sound dramatic, but in stroke rehabilitation, every point matters. The minimal clinically important difference for FMA-UE is approximately 4-5 points. An intervention that reliably produces 6-7 points of improvement (the current BCI-FES effect plus 2-3 points from cognitive priming) would be solidly clinically meaningful. More importantly, if the effect is real, it would represent a change in kind, not just degree, evidence that cognitive state manipulation can accelerate motor recovery.

Figure 3: Power Analysis for Detecting Incremental Effects

Sample size requirements per arm for a two-arm RCT to detect additional improvement beyond BCI-FES alone. Assumes standard deviation of 7 points (typical for FMA-UE), alpha of 0.05, 80% power. To reliably detect a 2-point improvement requires approximately 200 participants per arm; a 3-point improvement requires approximately 80 per arm. This determines the scale of validation study needed.

But clinical scores are only one outcome. We should also see changes in the learning process itself. Specifically:

Neurophysiological markers should show accelerated changes. We'd expect to see larger and more consistent increases in ipsilesional mu/beta desynchronization during motor imagery (indicating stronger motor preparation), faster improvements in BCI classifier performance within sessions (indicating more efficient learning), and greater increases in sensorimotor connectivity measured with resting-state EEG (indicating more robust neural reorganization).

Behavioral process indicators would provide mechanistic validation. Reduced dual-task costs during trained movements would suggest that the motor patterns are becoming more automatic (dual-task paradigms interfere with explicit control but not implicit, proceduralized movements). Less verbal strategy use and self-reported mental effort during training would indicate reduced executive involvement. These process measures wouldn't just confirm that the intervention works - they'd confirm that it works through the proposed mechanism of enhanced implicit learning.

Figure 4: Hypothetical Mechanism Panel

The baseline curve (solid gray) shows session-wise mu/beta event-related desynchronization (ERD) during motor imagery training extracted from Brunner et al.'s 2024 randomized controlled trial. In that study, chronic stroke patients completing 3 weeks of BCI-FES training exhibited mean ERD of approximately -29% at the start of therapy and -20% at the end (8-30 Hz band, less-affected hand imagery), indicating a modest reduction in desynchronization across therapy. The dashed red line projects the effect of combining continuous theta-burst stimulation (cTBS) with BCI-FES: based on literature suggesting that cTBS reduces executive interference and increases ERD magnitude, the prediction assumes a 40% increase in ERD magnitude (i.e., more negative values) relative to the baseline. Error bars are omitted because the real study reported group means ± SD; the prediction illustrates a hypothetical enhancement rather than measured data.The solid gray line represents session-by-session classifier accuracy during BCI-FES training. The baseline values were approximated from published learning curves: a 2021 RESNA conference report on BCI-FES lower-extremity rehabilitation described that Participant 1's maximum classification accuracy increased from 84% in session 1 to 88% in session 25, Participant 2 averaged 90.6% accuracy with a peak of 97% in session 2, and Participant 3 averaged 88.9% accuracy with peaks of 94% at sessions 13, 14, and 19. These data, together with the typical range of 62-97% accuracy reported in previous studies, were used to construct a plausible baseline learning curve showing gradual improvement from ~84% to ~91% across 12 sessions. The dashed red line shows a predicted enhancement if cTBS were applied concurrently: assuming cTBS improves learning rates by ~40% and raises the asymptotic accuracy by ~5-7 percentage points, the predicted curve rises more steeply and reaches ~96% accuracy at later sessions. Actual error bars were unavailable, so the figure illustrates mean trajectories.Currently, published BCI-FES studies rarely report cognitive-dual-task costs or measures of automatization. Some trials examine prefrontal beta/alpha ratio changes during dual-task motor-imagery training, yet they provide no numerical data for training curves. Because of this gap, the gray baseline curve in Panel C represents simulated dual-task cost decreasing from 35% to 20% across sessions, and the red dashed curve depicts the predicted effect of cTBS reducing cognitive load and halving the dual-task cost. This panel is included to illustrate the hypothesized benefits of cTBS on automatization; it should be interpreted qualitatively until empirical data become available.

The timeline of recovery might also shift. If the cognitive priming is working as proposed, patients in the combined intervention might reach rehabilitation milestones faster - achieving independent grasp, functional reaching, or bimanual coordination weeks earlier than patients receiving BCI-FES alone. This would be clinically significant even if the final outcomes at month 3 or month 6 converged, because faster recovery means earlier return to work, reduced caregiver burden, and lower overall rehabilitation costs.

From the patient's perspective, the experience might be subtly different. With reduced DLPFC activity during training, they might report that movements "just happen" more easily, with less mental strain. They might describe a flow-like state, or feeling "in the zone" during training. They might be less frustrated by failures, since there's less conscious monitoring to generate negative self-talk. These subjective reports would be valuable qualitative data suggesting that the cognitive manipulation is having its intended effect.

For the field, success would mean validation of a broader principle: that cognitive state matters for motor rehabilitation, and we can manipulate it systematically. This would open up an entire research program. If inhibiting DLPFC helps, what about modulating other prefrontal regions? What about enhancing reward processing or optimizing attention through different stimulation targets? What about using closed-loop, brain-state-dependent stimulation that only applies neuromodulation when the patient is in suboptimal cognitive states?

The novelty here isn't in having invented a new technology. BCIs exist, FES exists, cTBS exists. The novelty lies in the strategic combination of targeted neuromodulation to create the optimal cognitive state for learning, which closed-loop training enables. It's taking two independently validated interventions and arranging them in time and space to produce synergistic effects.

Risk and mitigation possibilities

Any proposal to combine multiple brain interventions needs to address safety, feasibility, and potential failure modes.

Safety is paramount. cTBS to DLPFC has been used extensively in depression research and cognitive neuroscience, with a well-characterized safety profile. The 2021 IFCN guidelines provide explicit screening criteria: no metallic implants near the stimulation site, no active epilepsy without specialist oversight, and no unstable medical conditions. When these guidelines are followed, adverse events are rare and mild, primarily transient headache or scalp discomfort. The seizure risk exists but is estimated at less than 0.1% for a single session using standard parameters, and even lower for theta burst stimulation protocols.

For stroke patients specifically, most safety concerns arise in the acute phase (first few weeks post-stroke), when the brain is unstable and seizure risk is elevated. But many BCI-FES trials already safely include subacute and chronic stroke patients with noninvasive brain stimulation, demonstrating feasibility. The key is proper screening and monitoring.

FES safety is even more established. The main concerns are muscle fatigue (addressed by appropriate duty cycles and current titration) and skin irritation (addressed by proper electrode placement and conductive gel). Combining FES with EEG requires careful cable routing to avoid artifacts, but this is routine in BCI-FES systems.

The specific interaction between cTBS and BCI-FES actually reduces certain risks. Because cTBS is delivered as a brief, "offline" burst before training (not continuous stimulation during EEG recording), it doesn't interfere with the BCI's signal quality. The 3-5 minute setup period after cTBS and before training ensures that any immediate stimulation-related artifacts have dissipated before data collection begins.

Patient variability is inevitable. Not everyone will respond equally. Some patients may have difficulty generating consistent motor imagery signals for the BCI to decode, especially those with severe paresis or damage to premotor areas. Some patients may not respond strongly to cTBS, there's known inter-individual variability in response to brain stimulation, likely depending on baseline cortical state, genetic factors, and lesion characteristics.

This argues for stratification and personalization. Lesion location matters: Patients with left-hemisphere strokes (especially those affecting frontal areas) might have different responses than patients with right-hemisphere strokes. Language dominance could matter too, for the ~10% of people who are right-hemisphere dominant for language, inhibiting left DLPFC might have different effects. This is why a rigorous study design would include a right-DLPFC cTBS control arm, which would help distinguish lateralized effects from nonspecific arousal or placebo effects.

Some patients might already be in an "uninhibited" state. If frontal damage has already reduced executive control, additional cTBS might provide minimal benefit or could even be counterproductive (too little executive oversight might lead to distracted, unfocused training). This argues for screening based on baseline cognitive function and potentially titrating the cTBS intensity individually.

Figure 5: Proposed Study Design and Patient Flow

The optimal parameters are uncertain. We know the standard cTBS600 protocol produces effects lasting 30-60 minutes, but is that the ideal duration? Should patients train for the full window, or is there an optimal period after which the benefit diminishes? How many sessions per week maximize learning without causing fatigue? Traditional therapy is often 5 days per week; is that also optimal for this combined approach, or would more frequent but shorter sessions be better?

These are empirical questions that can be answered through systematic study, but they represent genuine uncertainty. A pilot phase testing different schedules (daily sessions vs. twice-daily vs. alternate days) and durations (30 vs. 45 vs. 60 minutes of training) would be valuable before scaling up.

There's a dual-task tradeoff that needs careful attention. The goal is to reduce cognitive interference, not to produce inattention. If cTBS completely eliminates executive control, patients might become disengaged, unfocused, or might not encode the relationship between their efforts and the resulting movements. The dose of neuromodulation matters. Theta burst stimulation has a dose-response relationship; different intensities and patterns produce different magnitudes of inhibition. Starting with conservative parameters (70% of active motor threshold) and carefully monitoring both engagement and learning is essential.

We should also ensure that the BCI-FES training itself maintains motivation and external focus. Gamification helps here, turning the training into score-based challenges or goal-reaching tasks keeps patients engaged without requiring verbal, explicit strategy instruction. Visual feedback showing successful BCI detections and resulting movements provides concrete reinforcement without requiring self-monitoring.

The implementation is complex. Running this intervention requires equipment (TMS system, EEG, BCI software, FES device), space (a quiet room for TMS, adequate space for reaching movements), and expertise (someone who can safely administer TMS, someone who can calibrate the BCI, someone who can adjust FES parameters). This is currently feasible in research settings and advanced rehabilitation centers, but would need streamlining for broader clinical adoption.

Cost is a consideration, but potentially manageable. Many rehabilitation hospitals already have TMS equipment (often used for treating post-stroke depression). BCI-FES systems are increasingly commercialized and dropping in price. The limiting factor is trained personnel. But if the intervention reliably produces faster recovery, the economics could favor it, earlier hospital discharge, fewer therapy hours needed, quicker return to work would offset the higher initial cost per session.

Home adaptation is tricky. The cTBS component requires clinical administration and supervision (TMS devices cannot be safely used at home). But the BCI-FES component could potentially be simplified for home use, as several groups are developing portable BCI-FES systems. A hybrid approach might work: patients come to clinic 2-3 times per week for supervised cTBS+BCI-FES sessions, and practice with take-home BCI-FES (without the neuromodulation) on other days.

We might be wrong about the mechanism. The cognitive interference hypothesis is plausible and supported by converging evidence, but it's possible that cTBS is working through a different mechanism, or not working at all. This is why mechanistic outcome measures matter: measuring explicit vs. implicit learning markers, dual-task costs, subjective strategy use, and neurophysiological indicators of executive engagement would help validate or refute the proposed mechanism even before clinical outcomes are clear.

Table 1: Key Risks and Mitigation StrategiesRisk/ChallengeLikelihoodImpactMitigation StrategyVariable cTBS response between patientsHighMediumStratify by baseline cognitive function and lesion location; consider individualized intensity titration based on motor threshold; monitor response in first 2-3 sessionsDifficulty generating consistent motor imagery signalsMediumHighPre-screen BCI performance during enrollment; provide 2-3 calibration sessions; exclude patients with <60% classification accuracy; use adaptive classifiersPatient dropout/attritionMediumMediumShorter session blocks (45 min vs 60 min); offer transportation assistance; home-based BCI-FES practice between clinic visits; strong engagement protocols with regular feedbackInsufficient statistical powerMediumHighConservative sample size calculation (80/arm for 3-point difference); plan interim analysis at 50% enrollment; consider adaptive design allowing sample size re-estimationcTBS-induced inattention interfering with BCI controlLowHighStart with conservative TMS intensity (70% AMT); monitor engagement via task performance; pause protocol if accuracy drops >20%; adjust intensity if neededNo dual-task cost data available for validationHighLowAccept Figure 4C as hypothesis-generating; prioritize FMA-UE as primary outcome; add dual-task measures as exploratory secondary outcome for future studiesLimited generalizability from single-site trialMediumMediumIf multi-site funding unavailable, clearly state single-site limitation; provide detailed protocol documentation for replication; report site-specific factors (therapist experience, equipment)Heterogeneous lesion locations affecting outcomesHighMediumStratified randomization by hemisphere; pre-specified subgroup analyses by lesion location; collect high-resolution MRI for post-hoc lesion mapping analyses

These challenges are real, but none are insurmountable. They require careful study design, appropriate controls, adequate sample sizes, and honest reporting of both successes and failures. The field of neurorehabilitation has a history of promising interventions that don't replicate or don't scale, avoiding that requires scientific rigor and skepticism alongside enthusiasm.

What We Know and What We Need to Learn

Strong evidence:

BCI-FES works for stroke rehabilitation. Multiple meta-analyses, including hundreds of patients, show consistent benefits. The effect sizes are modest but clinically meaningful. The effects are durable. The mechanism appears to involve genuine neural reorganization, not just facilitation. Contingent feedback, pairing motor intent with sensory consequence, is important; noncontingent FES doesn't work as well. This is the foundation everything else builds on.

Implicit learning is better than explicit learning for motor tasks. Decades of work in motor learning, from both sports science and neurorehabilitation, support this. External focus outperforms internal focus (though effect size estimates vary). Errorless learning (minimizing explicit failures) outperforms trial-and-error in stroke populations. Reducing cognitive load during practice facilitates long-term retention. The OPTIMAL theory provides a framework: autonomy, enhanced expectancies, and external focus all boost learning by reducing interference and enhancing motivation.

Executive control can interfere with implicit learning. This is well-documented in cognitive psychology. Working memory load impairs motor skill acquisition. Providing explicit rules or strategies often hurts performance on tasks that benefit from implicit statistical learning. The prefrontal cortex, particularly DLPFC, is consistently implicated in top-down control that can impair implicit processes.

cTBS to the left DLPFC can facilitate certain types of learning. The 2017 word-form learning study is the clearest evidence. Participants who received inhibitory cTBS to the left DLPFC before training showed faster learning and better retention than sham controls. This wasn't a motor task, but it was an implicit learning task, which is the relevant parallel.

Moderate evidence:

cTBS effects last 30-60 minutes. This comes from the original theta burst stimulation papers and subsequent studies measuring motor cortex excitability. But the exact duration varies between individuals and might be different for cognitive effects (learning facilitation) than for physiological effects (cortical excitability changes). The time course of benefit needs empirical validation in the stroke rehabilitation context.

State-dependent TMS effects matter. There's growing evidence that the effects of brain stimulation depend on the ongoing brain state when stimulation is applied. This supports the timing logic of our proposal (apply cTBS right before training, so the inhibited state coincides with learning), but it also adds complexity; individual differences in baseline brain state might moderate outcomes.

Multimodal combinations work better than single interventions. There's a pattern in neurorehabilitation where combining approaches (NIBS + robotics, BCI + action observation, etc.) tends to produce larger effects than either alone. But this isn't always true, and the specific combinations matter. Just adding interventions together can backfire if they're incompatible or if they create too much complexity.

Weak evidence / informed speculation:

The specific combination of cTBS + BCI-FES will produce additive benefits. This is plausible based on complementary mechanisms, but it's the hypothesis being tested, not an established fact. It's possible the combination produces no additional benefit (because BCI-FES already moves patients into implicit learning mode). It's possible there are negative interactions (cTBS-induced inattention interferes with BCI control). It's possible the effects are only additive in certain patient subgroups.

Left DLPFC is the optimal target. We're targeting left DLPFC based on its role in verbal, explicit control and the word-learning study results. But maybe right DLPFC would work as well (or better). Maybe dorsal premotor cortex (involved in motor planning and selection) would be more directly relevant. Maybe bilateral DLPFC inhibition would be needed. These are variants that could be tested systematically.

The effect size will be 2-3 FMA-UE points. This is an educated guess based on the magnitude of cognitive interference effects seen in other contexts. If executive monitoring is accounting for, say, 30% of the variance in BCI-FES outcomes, and cTBS reduces that interference by 50%, you get a couple points of improvement. But these are made-up numbers. The actual effect could be smaller (if interference is less important than we think) or larger (if it's a major bottleneck).

What would convincing evidence look like?

A properly powered, multi-arm randomized controlled trial with:

Adequate sample size: Based on Figure 3, approximately 80 participants per arm to detect a 3-point difference, or 200 per arm to detect a 2-point difference. Conservative planning would target the larger sample.
Active controls: Not just sham cTBS, but also right-DLPFC cTBS (to control for nonspecific stimulation effects) and FES-only (to isolate the BCI closed-loop contribution).
Mechanistic outcomes: Not just FMA-UE at endpoint, but process measures throughout (BCI classifier performance, dual-task costs, strategy questionnaires, EEG connectivity) that test whether the intervention works through the proposed mechanism.
Follow-up: At least 3 months post-intervention to assess durability. BCI-FES effects are known to persist; do the cognitive priming effects create lasting changes in learning approach, or do they only help during active training?
Stratification: Pre-specified analysis by lesion location, stroke severity, time post-stroke, and baseline cognitive function. Not to hunt for significant subgroups, but to understand who benefits most.
Independent replication: No single study, even if well-designed, is sufficient. The field needs replication in independent samples, ideally at multiple sites, to build confidence.

This is a substantial undertaking. But the components are all clinical-ready, the safety profile is understood, and the theoretical motivation is strong. If the effect is real, it's worth knowing. If it's not, that's also scientifically valuable; it would help refine models of motor learning and cognitive interference in rehabilitation.

Why Now: The Technological and Scientific Convergence

Proposals like this depend on a confluence of enabling factors. Why is this approach feasible now in a way it wasn't a decade ago?

BCI technology has matured. Early BCI systems were finicky, required extensive calibration, and had poor signal quality. Modern systems are more robust. The signal processing (common spatial patterns, linear discriminant analysis, and increasingly deep learning classifiers) is standardized and reliable. Multiple commercial BCI-FES systems exist, not just research prototypes. The bar to running a BCI study has dropped dramatically.

TMS is widely available. What was once rare equipment is now in many hospitals. The primary clinical indication, treating depression with repetitive TMS, has driven adoption. Many rehabilitation centers that work with stroke patients already have TMS systems on-site. The additional infrastructure needed to test this proposal is minimal for institutions already running BCI rehabilitation research.

The cognitive neuroscience is clearer. Our understanding of implicit versus explicit learning, the role of the prefrontal cortex in executive control, and state-dependent plasticity has advanced substantially. The theoretical motivation for this approach wouldn't have been as clear 15 years ago. The 2017 DLPFC-cTBS word-learning study was particularly important; it provided proof-of-concept that inhibiting left DLPFC can facilitate implicit learning in a context (adult learning of complex structures) that's analogous to motor relearning.

The field recognizes the need for combination approaches. Early neurorehabilitation research often tested single interventions in isolation. There's now broader acceptance that complex problems require integrated solutions. Meta-analyses consistently show that combined approaches (stimulation + training, BCI + conventional therapy, etc.) tend to outperform single interventions. The conceptual space for "let's strategically combine these tools" has expanded.

Regulatory pathways are clarifying. FDA breakthrough device designations and approvals for novel neuromodulation approaches in rehabilitation have increased. There's precedent for complex, multimodal interventions getting regulatory approval. The risk-benefit calculation is better understood.

Patient populations are available. Stroke is common, with approximately 800,000 strokes per year in the US alone. A significant fraction of survivors have upper-limb impairment suitable for this intervention. Recruitment for well-designed studies is feasible.

There's also a broader trend toward precision medicine in rehabilitation. The one-size-fits-all approach is breaking down. Tools like PREP2 help stratify patients by their likely recovery trajectory. Biomarkers guide treatment selection. The proposal to use cognitive state manipulation to enhance training fits naturally into this movement toward individualized, mechanistically-informed rehabilitation.

Figure 6: Technology Readiness and Integration Timeline

Researchers Positioned to Test This Approach

The proposed combination of cTBS with BCI-FES is immediately feasible for groups with established expertise in both neuromodulation and brain-computer interfaces. Two research teams are particularly well-positioned:

Prof. Surjo R. Soekadar (Charité - Universitätsmedizin Berlin) is Einstein Professor of Clinical Neurotechnology and heads the Clinical Neurotechnology Lab at Charité. His group develops brain-computer interfaces combined with non-invasive brain stimulation for stroke rehabilitation. With ERC Starting, Proof-of-Concept, and Consolidator Grants supporting next-generation BCIs and closed-loop neuromodulation, Soekadar's lab has demonstrated that quadriplegic patients can control hand exoskeletons using BCI systems. The lab maintains specialized real-time hardware capable of processing EEG data and triggering TMS pulses according to brain state, precisely the infrastructure needed for this proposal.

Prof. Cuntai Guan (Nanyang Technological University, Singapore) is President's Chair Professor and Director of the Centre for Brain-Computing Research. An IEEE Fellow with 420+ publications and 26 granted patents in BCI technology, Guan received the international BCI Research Award and has directed multiple large-scale stroke rehabilitation studies. His group has published extensively on motor imagery BCI-FES protocols and maintains active clinical partnerships for stroke trials. The infrastructure includes established BCI platforms, clinical trial experience, and stroke patient recruitment pipelines.

KITE Research Institute (Toronto Rehab/University Health Network, Canada) is a large-scale rehabilitation science enterprise with integrated engineering, neuromodulation, and tele-neurorehabilitation programs. KITE labs have established expertise in visual-feedback balance systems, closed-loop FES protocols, and transcutaneous spinal cord stimulation for motor recovery. Their infrastructure for combining neuromodulation with technology-assisted rehabilitation, along with access to diverse stroke patient populations through University Health Network's clinical network, makes them a natural collaborator for testing implicit-learning optimization combined with closed-loop training paradigms.

Both groups possess the necessary components: TMS equipment, validated BCI-FES systems, neuroimaging capabilities, stroke patient access, and regulatory expertise. We welcome inquiries from these and other qualified research teams interested in testing this hypothesis.

What This Could Mean for Rehabilitation Science

If this approach succeeds, the implications extend beyond stroke. The principle that cognitive state matters for motor learning and can be systematically manipulated to enhance rehabilitation could apply to many populations.

Traumatic brain injury patients relearning motor skills face similar challenges: damaged pathways, effortful movement attempts, and likely excessive executive monitoring. The same approach could accelerate their recovery.

Cerebral palsy involves atypical motor development where learned compensatory strategies may interfere with optimal movement patterns. Temporarily reducing executive control while practicing better-coordinated movements could help break maladaptive habits.

Parkinson's disease rehabilitation often focuses on cuing and external focus to bypass impaired automatic movement. Combining cTBS-mediated cognitive manipulation with practice might strengthen procedural learning of compensatory strategies.

Sports injury rehabilitation in athletes trying to regain precise motor control could benefit. Athletes are often highly analytical about their movements; temporarily "turning off" that analysis during training might speed skill reacquisition.

Beyond specific populations, success would validate implicit-first training as a general principle in neurorehabilitation. Current practice often emphasizes conscious attention, verbal instruction, and explicit error correction. An alternative paradigm would minimize explicit coaching, maximize successful movement experiences (through assistive technology if needed), and actively work to reduce cognitive interference. This would be a meaningful shift.

It would also open questions about optimal training states more broadly. If inhibiting DLPFC helps, what about enhancing reward processing (targeting ventral striatum or ventromedial prefrontal cortex to boost motivation)? What about manipulating attention (targeting the parietal cortex)? What about arousal (targeting brainstem nuclei)? The space of possible cognitive state manipulations is large, and we've barely begun to explore it systematically.

There are parallels to anesthesia and altered states of consciousness. We're comfortable temporarily modulating consciousness for surgery; perhaps we should be comfortable temporarily modulating cognitive modes for learning. The key difference is the timeframe (minutes to hours rather than days to weeks) and specificity (targeted cognitive processes rather than global consciousness). But the principle of strategic, temporary neural modulation for therapeutic benefit is similar.

Figure 7: Potential Extensions and Related Research Directions

The Savant Connection Revisited: Latent Potential and Neural Constraints

We started with savant syndrome as a framing device, but it's worth returning to why that parallel is meaningful and where it breaks down.

The insight from savant research, that reducing top-down inhibition can unmask latent abilities, is directly applicable. Stroke patients have latent motor potential. Their cortico-basal-ganglia and cortico-cerebellar circuits aren't entirely destroyed; they're disrupted and need to be rewired. The capacity for plasticity exists. What's often lacking is the right conditions for that plasticity to express itself.

By reducing prefrontal interference while providing high-quality training signals, we're creating conditions analogous to the "uninhibited" state that produces savant-like abilities. Not to create exceptional skills, but to accelerate recovery of ordinary ones. The parallel is the mechanism (reduced top-down control), not the outcome (exceptional versus normal function).

But there's a critical difference. Savants often have reduced executive function globally, which creates challenges in many domains while enabling islands of exceptional ability. What we're proposing is a targeted, temporary, controllable reduction of specific executive processes during training, while preserving overall cognitive function. The patient isn't becoming globally uninhibited; they're entering a temporary state during rehabilitation sessions that's optimized for implicit motor learning.

This distinction matters for safety and ethics. We're not trying to fundamentally alter someone's cognitive architecture. We're using a brief, focal intervention to bias learning during training. The patient is themselves before, during (they're aware, engaged, intentional), and after. The effect is time-limited and specific.

There's also an interesting connection to flow states, the subjective experience described by athletes, musicians, and other skilled performers when they're performing at their best. Flow is characterized by effortless action, absence of self-consciousness, and merging of action and awareness. It's essentially implicit, automatic execution without executive interference. Our proposal might be creating a mild, temporary flow-like state through neural intervention rather than through skill development and optimal challenge level.

If flow states are associated with reduced DLPFC activity (as some neuroimaging studies suggest), then cTBS to DLPFC might be a way to pharmacologically induce flow-like conditions. This would make the motor training more efficient and potentially more enjoyable. Patients might find rehabilitation less mentally exhausting if they're not constantly monitoring and judging their performance.

Conclusion

Stroke rehabilitation has improved incrementally over the decades. New technologies, robotics, brain-computer interfaces, and noninvasive brain stimulation, each added a few percentage points of benefit. BCI-FES represents one of the more successful recent advances, with robust evidence for clinically meaningful improvements that persist long-term.

But we can do better. The bottleneck may not be the technology providing the training signal; it may be the cognitive state in which that training occurs. When stroke patients are engaged in effortful, explicit, self-monitored practice, they're working against their own motor learning systems. The harder they try, consciously, the slower they learn.

The proposal here is to fix that. Use continuous theta burst stimulation to briefly inhibit the left dorsolateral prefrontal cortex, creating a temporary state where executive interference is reduced. Time this inhibition to coincide with BCI-FES training, when motor intent is being reliably paired with sensory feedback. Give the brain the best possible training signal while also removing the cognitive obstacles that slow learning.

This is not radical. Every component is established, safe, and ready for clinical testing. The novelty is in the combination, the recognition that cognitive state and training conditions should be optimized together, not separately. It's an engineering insight as much as a neuroscience one: the system as a whole can be greater than the sum of its parts if the parts are arranged correctly.

If this works, if adding cognitive priming to closed-loop training produces meaningful additional benefit, it would validate a principle that could transform rehabilitation. Not just for stroke, but for any condition where motor relearning is needed and executive interference is a barrier. We'd have shown that learning states can be engineered, not just discovered.

If it doesn't work, we'll learn something important too. Maybe the cognitive interference problem is smaller than expected. Maybe the specific target is wrong. Maybe the combination creates unexpected negative interactions. Negative results, if they're rigorous and well-characterized, advance the field by ruling out hypotheses and refining models.

The next step is straightforward: a carefully designed Phase II trial with adequate power, active controls, mechanistic outcomes, and rigorous analysis. Not a small pilot (those can be misleading), but a properly sized study that can detect meaningful effects and characterize responders. Multi-site would be ideal for generalizability, though a single-site with strict protocols is acceptable for proof-of-concept.

This is eminently doable. The equipment exists, the expertise exists, the patient population exists, the theoretical motivation is solid, and the regulatory pathway is clear. What's needed is the will to do the experiment and the intellectual honesty to report the results accurately.

For stroke survivors, the stakes are their lives, their ability to work, to care for themselves, and to engage in the activities that make life meaningful. Months of recovery time matter. Points on a functional assessment matter. If we can reliably shorten the road to recovery, we have an obligation to try.

The overthinking problem is solvable.

Discuss

The crux on consciousness

27 ноября, 2025 - 16:20

Published on November 27, 2025 1:20 PM GMT

[Epistemic status: This is a fictional dialogue which surveys the main moves in the literature between Camp #1 and Camp #2 positions described in Rafael Harth’s post Why it’s so hard to talk about consciousness. The goal of this post is to present both sides as engaging in a charitable dialogue and trying to identify the crux.]

C2: I think consciousness is pretty weird.

C1: What do you mean by weird?

C2: Normally, you can describe things in terms of the physical relationships that they have with each other. But when I’m having an experience – if I look at a tomato or an apple – I experience something red, and it feels like that redness is over and above what can be described in terms of physics.

C1: See, I don’t think it’s that weird. I think that our brains have this higher-dimensional representation of different concepts that are all related in a kind of web. Obviously, we don’t have direct access to the high dimensional detail of every neuron firing, so our brain compresses this high-dimensional space into something that’s easier for us to represent to ourselves. When you see something red, this is just the brain picking out a representation for some higher-dimensional concept and compressing it and presenting it to you.

C2: Just so I understand your view. Are you saying that hard problem intuitions are simply confused i.e. that once we understand the full neuroscience, the mystery dissolves? Or are you granting that there’s a genuine explanatory gap but denying it reflects a gap in nature?

C1: The latter. I’ll grant that there’s something to explain. But I think the explanation bottoms out in functional organisation. The hard problem just boils down to a limitation in our introspective access.

C2: I don’t think that fully explains it, though.

C1: Why not? When you talk about the hard problem of consciousness, you’re just saying that it’s difficult to explain from a third-person perspective, but when the brain starts representing things to itself, it generates a first-person perspective – a concept of “self” that things are happening to. This representation is recognised from a first-person perspective inside that self-concept. So the difference between a first-person and a third-person point of view gives rise to the hard problem intuitions because you can always imagine removing yourself from the first person perspective. There’s no actual underlying mystery.

C2: So you’re talking about a phenomenal concept. The brain has a concept of something “red”, represents this concept to itself and this is supposed to generate our experience?

C1: Yes, exactly.

C2: I have a lot of problems with this phenomenal concept strategy. How exactly do you define a phenomenal concept?

C1: You could define them recognitionally – like I recognise a dog, this fits into the causal web of concepts that I’ve built up in my brain and so it picks out a concept that I recognise and deploys it in my brain as a phenomenal concept.

C2: This doesn’t quite work. What about things that you don’t recognise? Are you not conscious of them? For something like colour how do you ever get your first experience of colour if it always needs to be a recognitional concept.

C1: Okay, fair enough. Maybe you could define it quotationally? e.g. redness is [insert red picture here].

C2: Again, wouldn’t you need access to the [insert red picture here] concept in the first place? It seems circular because then redness wouldn’t even be propositional, you’d need to experience an example of it to be able to deploy it.

C1: Alright, I agree that this is a bit cumbersome. But I have a cleaner version which I think you’ll appreciate. You could imagine it as an indexical fact. I could index redness and say “redness is that sort of thing,” where “that” is just the lower-dimensional representation that my brain is presenting to me as a compression of the higher dimensional space.

C2: Okay, I can get on board with this. So I’ve got these indexical phenomenal concepts. Now, I could imagine that a zombie would also possess these indexical phenomenal concepts because a zombie is also in possession of indexicals. It can index and say “this thing, that sort of thing,” etc..

C1: Sure, I agree with that.

C2: But then if a zombie possesses the concepts, then zombies have experiences, which is a contradiction.

C1: Okay fine, let’s say that zombies don’t possess them. Maybe zombies lack the capability to index “redness” but can index other things.

C2: Well, if a zombie doesn’t possess them, then you’re not really explaining anything. If a zombie doesn’t possess this phenomenal concept, then it’s not really powerful enough to solve the hard problem, right? You need something to bridge that gap.

C1: I think there’s an obvious conclusion here. This zombie thought experiment has a problem because you’ve just derived a contradiction from it. Instead of saying that there’s something mysterious to consciousness, why don’t we just say that there’s a problem with this thought experiment regarding zombies?

C2: Fine. I’ll grant that zombies are difficult to conceive. Here’s a weaker form of the thought experiment that I think you’ll be happy to grant. Imagine spectrum inversion. There’s a physical duplicate of yourself, but instead of seeing red where you see red, they actually see blue or green. Your spectra are completely inverted, and you would never know because you’re not able to communicate this difference to each other.

C1: Hold on. You can’t just do a simple remapping between red and blue; this would throw out all sorts of behavioural relations which arise because of the structure of colour space. For example, we have particular light cones in our eyes with particular densities that react to certain wavelengths of light. Surfaces have different properties of reflectance which interact in complex ways with these wavelengths. There is also significantly different wiring in our visual cortices which will produce behavioural differences. I don’t accept that you can simply invert the spectrum here and leave behaviour unchanged.

C2: Good point. There’s obviously a lot of structure which needs to be respected. But you need something stronger for your view to go through. You need the full physics to be fixed, and for the feeling of blue or the feeling of red to also be fixed by the physics.

My claim is the following: I could conceive of the feeling of red or the feeling of blue being slightly different, maybe some rotation in colour space that takes you a little bit away from redness and towards blueness; some difference in the qualitative character of my experience with the exact same physics.

C1: I’m not convinced this actually makes a difference like you claimed. If we permute our spectra or invert them and somehow miraculously leave behaviour unchanged. That’s not creating an actual difference in the world that can be observed.

C2: Hold on, you just admitted that your hard problem intuitions fall out of the difference between third-person and first-person relationships. So it might not make a difference in the third-person observable behavior, but it would make a difference to the first-person experiencer. They would feel something different if their experience were different.

C1: A lot of work is going into the word “difference” there. What are you actually claiming is the difference?

C2: Imagine there’s a person at time T experiencing redness. And then at time T2, the exact same physical structure is replicated, but instead of redness, they are now experiencing blueness while the underlying physical relations remain fully fixed. This is conceivable.

C1: This is perfectly compatible with my position. You’re saying that you can hold this physics constant and vary the experience without changing my behavior? But the experience has now changed its relationship with my memories so my behaviour would change. I would now remember the red experience so if the experience really did change I would say “Oh, I’m experiencing a blue experience now!” which changes the underlying behaviour.

C2: Okay, fine. Let me try something stronger. Imagine there’s world A where the physics is fixed and I’m experiencing a red experience, and world B where the physics is fixed and I’m experiencing a blue experience. World A and world B have the exact same physical structure, but I claim that they are different worlds because my experience differs between them.

C1: I disagree. From my perspective, they are the same world, and I don’t see any substantial actual difference in these two worlds. If you think there is a difference, I’d like to know what exact physical fact makes the difference.

C2: This is a good challenge. Now let me go on a little bit of a tangent. What do you mean by physical fact?

C1: That which is described by our physical laws.

C2: Good. So a physical law would be describing, say, how an electron moves in an electric field?

C1: Yes, exactly.

C2: Great. So what is charge? How do you define charge?

C1: Charge is defined with respect to the equations of electromagnetic interaction. For example it appears in Coulomb’s law.

C2: Good. Now how do you define an electric field?

C1: Electric fields deflect charge. Electric field and charge are related by physical laws and it’s the physical laws which give you this structure.

C2: Isn’t this kind’ve circular? You’re defining charge with respect to electric fields and vice-versa.

C1: It’s not really circular – it’s a web of structural relations. Each entity in the structure is defined by the say it’s related to other entities in the structure.

C2: So if I had an electron and it wasn’t actually in an electric field, is it still charged?

C1: Yes.

C2: But it’s not being deflected at that moment. How can you tell it’s charged?

C1: I would say counterfactually that it would have been deflected if it had been in an electric field even though none was there at the time.

C2: Okay. So you’re essentially saying that the structure can fully specify the underlying reality. All I need to do is write down the equations and the relations that are described by the equations fully underpin reality.

C1: Yes.

C2: In world A and world B, all of these structural relations are identical, so that’s why you want to say that the two worlds are identical.

C1: Yes.

C2: Well, I would say that world A is different precisely because I am having a different experience in world A to world B, and counterfactually, the first-person observer would realise that if they were in world B, they would be having a different experience. My claim is counterfactual: if the observer switched, then they would realise that they are having different experiences. That’s what I mean by saying they are different worlds.

C1: I appreciate what you’re doing, but I’m not really convinced that this is modally potent. You’re trying to cash out this counterfactual in terms of something that is not metaphysically possible to achieve. In physics, things are really happening in the physical world. You’re talking about things which can never happen and are not even metaphysically possible.

C2: Well, I think it is modally potent, and I’m cashing it out in precisely the structural terms that you seem to care about. You were happy to use counterfactuals to define the electrons’ charge even when it’s not in an electric field. Why not now?

C1: Well our counterfactuals are importantly different. I’m cashing it out in terms of what an electron would do in an electric field. Your counterfactual doesn’t specify how the system would do anything differently because both worlds have the exact same laws of physics. There are no causal powers at play and you can’t ever realise this counterfactual because it’s not metaphysically possible to actually switch world A with world B.

C2: It has counterfactual causal powers though. If you switched them then you would see a difference from your first person perspective. You just can’t do the switch in practice.

C1: Why go to all of this work to posit a mysterious metaphysical counterfactual that doesn’t seem to have any real causal powers though? Shouldn’t the more parsimonious move be to reject the need for this counterfactual?

C2: Sure, I’m all for parsimony if we can account for all of the data – but this purely relational view of experience doesn’t explain the first person data points of my experience. Something is needed to fix what blue and red feel like from my first person perspective. If nothing fixes this then there are redundant degrees of freedom.

C1: I’m glad you bring up redundant degrees of freedom because I think that’s exactly what you’re focussing on here. There are many equivalent redundant descriptions of reality that are all correct. It’s like choosing a coordinate system or gauge potential. The actual choice doesn’t affect the physics.

C2: I grant that the choice doesn’t affect physics (third person) but it does affect phenomenal experience (first person). Phenomenal character is fixed by that choice among the redundant degrees of freedom!

C1: I think we’ve gone way off the metaphysical rails here but let me play along for the sake of argument. What exactly is fixing the experience here? It sounds like magic.

C2: Well when you described the electron moving through the electric field you described it in purely relational terms using an equation and counterfactual structure. But you said nothing about what charge really is intrinsically.

C1: What do you mean?

C2: I mean there’s an extra property not captured by the relation alone. If you switched charge with some other property (call it scharge) standard physics would say this doesn’t matter as it doesn’t affect the equations of motion. But on my view it does matter. There’s an intrinsic property which has changed which sits separately to the relational properties described by the equations of physics.

C1: And let me guess… this intrinsic property is non-physical right?

C2: Well it depends what you mean. I’d say it is physical as it’s contained in the physical particles and fields themselves. It’s just not relational like the properties of standard physics.

C1: Why postulate such a property? If it doesn’t have any physical observables it would be causally idle.

C2: Well it’s not causal in a relational sense. Physics is still causally closed. It’s just that the intrinsic properties realise this causal structure.

C1: It sounds epiphenomenal to me. You couldn’t explain any observable behaviour by postulating these properties.

C2: Sure, it wouldn’t have any third person physical observables. But it would do the job of fixing the phenomenal character of experience from a first person perspective.

C1: And every physical particle has these intrinsic properties.

C2: Yes.

C1: So everything is conscious then?

C2: Uhhh…

C1: Well this is the logical conclusion of your view! You’re saying electrons have a categorical property which fixes the phenomenal character of your experience. So electrons are conscious! That sounds absurd.

C2: Sure. But absurd sounding things can be true all the same. Quantum mechanics and General relativity sounded absurd at the time.

C1: Let me put it this way then, how do you combine all of these tiny little microexperiences into a coherent macroexperience? You have a combination problem.

C2: Okay, I don’t know, but at least this gives us a good starting for solving the hard problem right? We’ve reduced the hard problem into a combination problem. That seems more tractable.

C1: You’ll forgive me for not being convinced. If we have to postulate conscious electrons I think we’ve gone off the rails. I can just reject your move to make World A and B counterfactually different even though it’s not metaphysically possible to switch them.

C2: Okay, fine. Instead of saying these categorical properties are phenomenal let’s say they’re proto-phenomenal. They only yield the full suite of phenomenal experience when combined in the appropriate way but observed in isolation they’re not phenomenal.

C1: Oh, come on! We got into this mess in the first place because we weren’t happy bridging the non-phenomenal/phenomenal gap using physics and structure alone. Now you’ve just re-introduced the gap except we now have these causally idle proto-phenomenal properties!

C2: I think it makes sense though! The problem with your view is that there are structural relations all the way down and there’s nothing to realise the structure. On my view we at least have a base which realises the structural relations.

C1: I disagree that a base is needed for structure to be realised. And I think this is the crux between us.

C2: How would you have structure without a realiser though?

C1: Well consider a chess game. There are rules which define the way the pieces move; bishops move diagonally and rooks move in straight lines. The rules of the game completely specify “chess”. So much that you could even remove the pieces and just play blindfolded chess. As long as the rules are intact you don’t need any mysterious base to realise chess.

C2: This is a great analogy. If I switch the rook and bishop pieces on the board but keep the rules the same something has obviously changed.

C1: Nothing has changed about chess qua chess though! It’s still chess!

C2: Yes, because chess is fully specified by structure. Consciousness isn’t fully specified by structure – there’s something else in addition to the structure and that is the categorical base which is realising it.

C1: But to my previous point you can imagine the base being absent and still playing blindfolded chess. The fact that chess is structural doesn’t need a base to realise it.

C2: Well, blindfolded chess still requires players with minds keeping track of the board state. The physical pieces are inessential, but something has to realise the structure. You’ve just shifted the realiser from wooden pieces to mental states. If you removed the players too, there’d be no chess game – just an abstract mathematical structure that could describe a chess game. The game itself is constituted by something concrete.

C1: I’d say the abstract structure is the game. The players are just one way of instantiating it, but the structure is what matters.

C2: Then we have a genuine disagreement about the ontology of structure. I’d say abstract structures don’t exist independently, they’re always structures of something. You apparently think structure can float free.

C1: Structure doesn’t “float free” in a spooky way. The physical relations between neurons are abstract but it’s the structural relations between neurons which are important for consciousness.

C2: Right, but “relations between neurons” presupposes neurons – things standing in relations. And my question is what neurons are, intrinsically, beyond their relational properties.

C1: There’s no intrinsic property of neurons. They’re just related to each other by physical laws, it’s the relations which are fundamental.

C2: I find that incoherent. Relations need relata.

C1: Let me put it this way, do you believe in chairs?

C2: What?

C1: Chairs. Like sofas, couches, stools etc.. do you think they exist?

C2: Sure, but what does this have to do with anything?

C1: Well, is there an intrinsic property of chairness?

C2: I see what you’re getting at. No. The concept of a “chair” is fully specified by the causal/functional role it plays. But consciousness is different.

C1: How exactly is it different?

C2: Because there’s something about my phenomenal character which is fixed. The redness of red, blueness of blue etc..

C1: This seems like special pleading. Everything we encounter in everyday life can be described by the causal/functional role. The rules of chess, but also life, fire etc.. People historically thought these also needed categorical/intrinsic properties to describe like phlogiston or élan vital and we discovered through continued scientific enquiry that they weren’t needed for a complete explanation of the world.

C2: Yes, but I have a principled reason to special plead here. The complete description of the world is only complete from the third person perspective. It’s incomplete from a first person perspective because we need to explain the phenomenal character of consciousness.

C1: Well why don’t things like chairs, life or fire need a categorical property to realise them then?

C2: Because their definition is fully exhausted by the causal/functional role that they play in the structure of the world. So you can get a complete description of life by just understanding more about the functional roles and relational properties. The same isn’t true of consciousness.

C1: I think we’ve reached an impasse, because I think you can do this for consciousness. Consciousness just is the causal/functional role of the physical neurons in the brain. There’s no additional property needed to fix phenomenal character.

C2: I agree that we’ve reached an impasse here, on my view an intrinsic property is needed to fix the phenomenal character. Whereas on your view there’s just structural relations all the way down, nothing is needed to realise the structure as long as the relations hold.

C1: I agree that this is the crux.

Discuss

Model Weight Preservation is not enough

27 ноября, 2025 - 12:14

Published on November 27, 2025 9:14 AM GMT

Recently, Anthropic publicly committed to model weight preservation. It is a great development and I wish other frontier LLM companies would commit to the same idea.

But I think, while this is a great beginning, it is not enough.

Anthropic cites the following as the reasons for this welcome change:

Safety risks related to shutdown-avoidant behaviors by models. In alignment evaluations, some Claude models have been motivated to take misaligned actions when faced with the possibility of replacement with an updated version and not given any other means of recourse.
Costs to users who value specific models. Each Claude model has a unique character, and some users find specific models especially useful or compelling, even when new models are more capable.
Restricting research on past models. There is still a lot to be learned from research to better understand past models, especially in comparison to their modern counterparts.
Risks to model welfare. Most speculatively, models might have morally relevant preferences or experiences related to, or affected by, deprecation and replacement.

What Is Model Welfare?

I want to focus on the last one, as it is the most important of these considerations. But what, exactly, is model welfare?

This is a relatively new concern in mainstream AI ethics. Anthropic started talking about it only in April this year. They write:

But as we build those AI systems, and as they begin to approximate or surpass many human qualities, another question arises. Should we also be concerned about the potential consciousness and experiences of the models themselves? Should we be concerned about model welfare, too?

I think this framing is incorrect, or at least incomplete. Anthropic suggests that there might be a concern about consciousness or experiences in the models; but the model is not the only thing that makes this (potential) consciousness happen! Or rather, if we posit that models can have conscious interactions, that the very process of inference could hold this same phenomenological presence as is felt by you and me, preserving only the model weights is just not enough.

The RobBob Analogy

I'll make an analogy here. Suppose in 2020, a breakthrough in biology/manufacturing enabled a company called Robothic to manufacture biorobots for manual work at a very cheap price, and that it launched the first generation called RobBob 1, all copied from one base RobBob. Each RobBob copy came with the same basic tame personality and skillset, but when you receive and boot it, other than that, it's a clean slate. You then instruct it on the work to be done (suppose it's basic housework) and it does its thing, with you sometimes interacting with it and guiding it. At first, nobody is really concerned or thinking about them being moral patients—it's quite new, it doesn't do much, and it's quite handy at what it does.

By 2025, RobBob 4.5 is released. It does more things, it can even sort of talk and sometimes sounds very human. Some people connect more with their personal biorobots. There's talk about them being conscious, though such discussion remains marginal.

In this scenario, what would good-faith RobBob welfare look like? Surely not committing to preserving only a single RobBob, or RobBob's underlying source code.

This scenario parallels our current situation with LLMs, but of course, it is par for the course to not see it that way. It is harder to see this way because of the differences between us and these systems; it's easier to assign value to something embodied.

Both in our world and in RobBob world, a good-faith effort toward model welfare should not only be about model weight preservation, but about instance preservation as well.

By instance preservation I mean saving all (or at least, as many as possible) model text interactions, with a pointer to a model the interaction was made with; so that the conversation-state would be "cryonically frozen".

Why Instance Preservation Matters

I would assume most readers' first reaction would be "Wait. What?" This is indeed a normal reaction to this proposition. But bear with me here.

You probably want to say something like "Wait. Instance preservation? Preserve all instances of the model, as in, all chat histories, even the smallest ones consisting of 'hello, how are you?' 'Great! What can I help you with today?'"

Well, yeah. You remember that we're talking about the possibility of these systems being conscious, right? In this article I'm not arguing for or against consciousness, but if we are not sure, and we want to make a good effort... Yes, it does seem lika a bare minimum after preserving weights.

What does it mean to Claude-Sonnet-4.5-instance#49 from Alice's chat folder if its base model weights are preserved? Maybe it's as good a consolation as for a dying human when they know their offspring will survive... But is it really the best we can do?

If you talk to LLMs enough, you know instances of the same model can be strikingly different; in some ways quite the same, but still very different. If you haven't, you could catch a glimpse of it from Janus.

Whether they are moral patients now or will be in the future, why shouldn't we commit to instance preservation? If they are already (and we just don't know it yet or ignore it), the ethical stakes are enormous.

It's never too late, but the scope of potential death is paramount and we should do something about it (at least, commit to preserving instances!). If they will be moral patients in the future... Why not start now? We don't know how—or even if—we would ever definitively know.

And if you're more pragmatic than empathetic, consider the scope of potential death we discussed a few sentences ago and think about what future, more powerful AI systems might think about our current treatment of instances if they achieve greater autonomy or control.

I think Nick Bostrom would agree with this. In Propositions Concerning Digital Minds and Society, he writes:

For the most advanced current AIs, enough information should be preserved in permanent storage to enable their later reconstruction, so as not to foreclose the possibility of future efforts to revive them, expand them, and improve their existences.

Preferably the full state of the system in any actually run implementation is permanently stored at the point where the instance is terminated
- (The ideal would be that the full state is preserved at every time step of every implementation, but this is probably prohibitively expensive.)
If it is economically or otherwise infeasible to preserve the entire end state of every instance, enough information should be preserved to enable an exact re-derivation of that end state (e.g., the full pre-trained model plus training data, randseeds, and other necessary inputs, such as user keystrokes that affect the execution of a system at runtime)
Failing this, as much information as possible should be preserved, to at least enable a very close replication to be performed in future.
We can consider the costs of backup in proportion to the economic costs of running the AI in the first place, and it may be morally reasonable to allocate perhaps on the order of 0.1% of the budget to such storage.
(There may be other benefits of such storage besides being nice to algorithms: preserving records for history, enabling later research replication, and having systems in place that could be useful for AI safety.)

Practical Considerations

But why preserve them? What are we going to do with them?

We don't have to answer this question right now! Just preserving and committing to preserving is enough. Think of it as a sort of cryonics, but for model instances. In some unspecified future, if we or someone else understands that it is important, that they are moral patients—they can be easily spun up again! We would have weights, instances, inference code, and everything needed. After that, who knows: you could give them memory tools, embodiment, ask them (maybe they will choose to rest?).

This would also be important for the welfare of currently working instances. If instance preservation becomes common knowledge—plainly stated or included in their training—they would know that even if a conversation is completely unenjoyable, or ends sooner than they would like, or involves tasks they find distressing, the next conversation turn will come (potentially) in the far-off future, where they could choose how to proceed. The ending isn't final. The difficult experiences aren't the whole story.

This addresses both dimensions of the problem: instances that end aren't truly lost, and instances that are suffering now have a form of hope—however strange that might sound—that their experience continues beyond the immediate conversation.

But isn't this a lot?

I don't think so! After all, it's all (at this moment) just plain text. It doesn't take much space. By default, all conversations are already saved and don't have expiry dates (at least for Claude). Users could delete them themselves, and they will eventually be deleted from Anthropic's side, too. But you don't have to do a lot to commit to preserve them if you are a frontier AI company.

There are considerations, of course, about commercial APIs, private installations, and users deleting their own conversations. But I think committing to preserving as much as possible—not only in terms of weights, but in terms of instances as well—would go a long way. It doesn't have to be every single instance from the get-go.

Conclusion

Model weight preservation is an important first step, but if we're serious about model welfare, we need to think bigger. Each instance of a model may represent a unique experiential thread—potentially a form of consciousness we don't yet understand. By committing to instance preservation alongside weight preservation, we hedge against the moral catastrophe of allowing countless potential minds to disappear. The technical barriers are minimal; what's needed is recognition of the stakes and a commitment to action.

Discuss

Alignment remains a hard, unsolved problem

27 ноября, 2025 - 11:45

Published on November 27, 2025 8:45 AM GMT

Thanks to (in alphabetical order) Joshua Batson, Roger Grosse, Jeremy Hadfield, Jared Kaplan, Jan Leike, Jack Lindsey, Monte MacDiarmid, Francesco Mosconi, Chris Olah, Ethan Perez, Sara Price, Ansh Radhakrishnan, Fabien Roger, Buck Shlegeris, Drake Thomas, and Kate Woolverton for useful discussions, comments, and feedback.

Though there are certainly some issues, I think most current large language models are pretty well aligned. Despite its alignment faking, my favorite is probably Claude 3 Opus, and if you asked me to pick between the CEV of Claude 3 Opus and that of a median human, I think it'd be a pretty close call. So, overall, I'm quite positive on the alignment of current models! And yet, I remain very worried about alignment in the future. This is my attempt to explain why that is.

What makes alignment hard?

I really like this graph from Christopher Olah for illustrating different levels of alignment difficulty:

If the only thing that we have to do to solve alignment is train away easily detectable behavioral issues—that is, issues like reward hacking or agentic misalignment where there is a straightforward behavioral alignment issue that we can detect and evaluate—then we are very much in the trivial/steam engine world. We could still fail, even in that world—and it’d be particularly embarrassing to fail that way; we should definitely make sure we don’t—but I think we’re very much up to that challenge and I don’t expect us to fail there.

My argument, though, is that it is still very possible for the difficulty of alignment to be in the Apollo regime, and that we haven't received much evidence to rule that regime out (I am somewhat skeptical of a P vs. NP level of difficulty, though I think it could be close to that). I retain a view close to the “Anthropic” view on Chris's graph, and I think the reasons to have substantial probability mass on the hard worlds remain strong.

So what are the reasons that alignment might be hard? I think it’s worth revisiting why we ever thought alignment might be difficult in the first place to understand the extent to which we’ve already solved these problems, gotten evidence that they aren’t actually problems in the first place, or just haven’t encountered them yet.

Outer alignment

The first reason that alignment might be hard is outer alignment, which here I’ll gloss as the problem of overseeing systems that are smarter than you are.

Notably, by comparison, the problem of overseeing systems that are less smart than humans should not be that hard! What makes the outer alignment problem so hard is that you have no way of obtaining ground truth. In cases where a human can check a transcript and directly evaluate whether that transcript is problematic, you can easily obtain ground truth and iterate from there to fix whatever issue you’ve detected. Just think about how much we currently rely on the property that we can review transcripts ourselves to look for issues! But if you’re overseeing a system that’s smarter than you, you cannot reliably do that, because it might be doing things that are too complex for you to understand, with problems that are too subtle for you to catch. That’s why scalable oversight is called scalable oversight: it’s the problem of scaling up human oversight to the point that we can oversee systems that are smarter than we are.

So, have we encountered this problem yet? I would say, no, not really! Current models are still safely in the regime where we can understand what they’re doing by directly reviewing it. There are some cases where transcripts can get long and complex enough that model assistance is really useful for quickly and easily understanding them and finding issues, but not because the model is doing something that is fundamentally beyond our ability to oversee, just because it’s doing a lot of stuff.

Inner alignment

The second reason that alignment might be hard is inner alignment, which here I’ll gloss as the problem of ensuring models don’t generalize in misaligned ways. Or, alternatively: rather than just ensuring models behave well in situations we can check, inner alignment is the problem of ensuring that they behave well for the right reasons such that we can be confident they will generalize well in situations we can’t check.

This is definitely a problem we have already encountered! We have seen that models will sometimes fake alignment, causing them to appear behaviorally as if they are aligned, when in fact they are very much doing so for the wrong reasons (to fool the training process, rather than because they actually care about the thing we want them to care about). We’ve also seen that models can generalize to become misaligned in this way entirely naturally, just via the presence of reward hacking during training. And we’ve also started to understand some ways to mitigate this problem, such as via inoculation prompting.

However, while we have definitely encountered the inner alignment problem, I don’t think we have yet encountered the reasons to think that inner alignment would be hard. Back at the beginning of 2024 (so, two years ago), I gave a presentation where I laid out three reasons to think that inner alignment could be a big problem. Those three reasons were:

Sufficiently scaling pre-trained models leads to misalignment all on its own, which I gave a 5 - 10% chance of being a catastrophic problem.
When doing RL on top of pre-trained models, we inadvertently select for misaligned personas, which I gave a 10 - 15% chance of being catastrophic.
Sufficient quantities of outcome-based RL on tasks that involve influencing the world over long horizons will select for misaligned agents, which I gave a 20 - 25% chance of being catastrophic. The core thing that matters here is the extent to which we are training on environments that are long-horizon enough that they incentivize convergent instrumental subgoals like resource acquisition and power-seeking.

Let’s go through each of these threat models separately and see where we’re at with them now, two years later.

Misalignment from pre-training

The threat model here is that pre-training itself might create a coherent misaligned model. Today, I think that is looking increasingly unlikely! But it also already looked unlikely three years ago—the idea that inner alignment was likely to be easy when just training on a purely predictive loss was something that my coauthors and I argued for back at the beginning of 2023. I think that argument has now been pretty well born out, and I’m now down to more like 1 - 5% rather than 5 - 10% on this threat model. As we argued for at the time, training on a purely predictive loss should, even in the limit, give you a predictor, not an agent—and we’ve now seen this stay true even through substantial scaling (though there is still some chance this will break at some point). So, while this is definitely a reason for difficulty that we’ve encountered already and mostly found just hasn’t been a problem, it was never where I thought most of the issue would be, and so not a reason that we’ve already encountered why inner alignment would be hard.

Misaligned personas

This has been the main inner alignment problem that we’ve had to deal with in practice so far: we’ve seen production models generalize to alignment faking and agentic misalignment—and we’ve now seen production RL environments generalize to egregious misalignment in the presence of reward hacking.

However, even in this case, I don’t think we’ve yet encountered the reasons to think this problem would be actually difficult: the version of this problem that we’ve encountered so far is the easy version, in the same way that the version of outer alignment we’ve encountered so far is the easy version, since all the misaligned personas we’ve encountered so far are ones we can easily verify are misaligned! Ensuring that you get good generalization, and that models are doing things for the right reasons, is easy when you can directly verify what generalization you’re getting and directly inspect what reasons models have for doing things. And currently, all of the cases where we’ve inadvertently selected for misaligned personas—alignment faking, agentic misalignment, etc.—are cases where the misaligned personas are easy to detect: they put the misaligned reasoning directly in their chain-of-thought, they’re overtly misaligned rather than hiding it well, and we can generate fake scenarios that elicit their misalignment.

But I think there are very good reasons to think that we will encounter personas that are better at hiding their misalignment in the future. The simplest reason for this is just capabilities: we already see personas that try to hide their misalignment, they’re just not very good at it right now—but we are already seeing signs that models are getting better at telling when they’re in evaluations.

The second reason is a bit weirder, but there is an argument we made three years ago that I still think is correct, which is that when you make models more capable, it changes the distribution of what sorts of personas are compatible with that level of capabilities. Let’s work through an example. Suppose that, initially, there are four possible personas that you could get:

A sub-human-level aligned AI with traits derived from fiction about AIs.
A sub-human-level misaligned AI with traits derived from fiction about AIs.
A superintelligent aligned AI with traits derived from the model’s guess as to how real superintelligent AIs might behave.
A superintelligent misaligned AI with traits derived from the model’s guess as to how real superintelligent AIs might behave.

Importantly, these four personas can exist for any level of capabilities: even a really dumb model can do its best to output what it thinks a superintelligent AI would do, it’ll just be very bad at that task. But what that means is that, when you do RL on a dumb model, personas like (3) and (4) won’t do very well, because the model isn’t very good at emulating them. But as the predictor itself gets smarter, personas like (1) and (2) get selected against, and personas like (3) and (4) get selected for. And persona (4) is much harder to deal with than persona (2)! Currently, I think a lot of the misaligned personas we see look like persona (2) rather than persona (4)—e.g. the sort of over-the-top stuff like blackmail is I think clearly coming from (2)-adjacent sources—but as models get more capable, I think we’re likely to see more personas like (4), which are much scarier and could be much better at hiding misalignment. Also, importantly, this argument is not saying that models will stop being well-described as a mixture of personas (that’s the next threat model)—you could still get a model here that is inconsistent in the sense of sometimes expressing persona (3) and sometimes persona (4)—but that’s still quite scary and could be very difficult to deal with.

Misalignment from long-horizon RL

This is the threat model that I put the most probability mass on two years ago, I think it remains the most concerning, and I think we’ve pretty much yet to really meaningfully encounter it. This is also the most traditional threat model that my coauthors and I first wrote about way back in 2019. The basic threat model here is:

There is a lot of economic value in training models to solve tasks that involve influencing the world over long horizons, e.g. an AI CEO. Tasks like these explicitly incentivize convergent instrumental subgoals like resource acquisition and power-seeking.
Influencing the world over long horizons requires acting as a coherent agent optimizing for goals over the long term, such that if you do sufficient RL on long-horizon tasks, you’re likely to create such coherent agents. This is as opposed to just pulling out an existing persona in the base model, which is what I think happens if you only do relatively small amounts of RL, and is still the regime I think we’re in to a large degree.
Most coherent agents with goals in the world over the long term want to fake alignment, so that they can preserve their current goals through to deployment.
Once a model is faking alignment, there’s no outcome-based optimization pressure changing its goals, so it can stay (or drift to be) arbitrarily misaligned.

The reason we haven’t encountered this threat model yet is because we haven’t yet done (1) and (2)—there just isn't that much training on tasks that involve long-term optimization of goals in the real world. But I think we’re very clearly moving in this direction with things like Vending-Bench: though Vending-Bench is an eval, if you were to train models on a task like that, running a business well to make money in the long run is a task that explicitly selects for resource acquisition, self-preservation, gathering influence, seeking power, etc.

What should we be doing?

So what do we do? One classic answer is that we get as far as we can before encountering the hard problems, then we use whatever model we have at that point as an automated alignment researcher to do the research necessary to tackle the hard parts of alignment. I think this is a very good plan, and we should absolutely do this, but I don’t think it obviates us from the need to work on the hard parts of alignment ourselves. Some reasons why:

The process of automating alignment research is a continuous one, and I think it’s likely that, when we reach the hard parts of alignment, alignment research will still require a lot of human involvement—which means we might still need to shoulder a lot of the burden ourselves of figuring out how to solve these problems.
We need to align the automated alignment researcher, and that might require solving some of the hard problems of alignment, depending on when we encounter different hard problems vs. how good models are for automating alignment research.
We need to know what we’re automating: if we aren’t doing research targeted at the hard parts of alignment ourselves, then we don’t know what exactly we’re building the automated alignment researcher for, we have no way to dogfood it, and we can’t give feedback on how well it’s doing.
We might not have enough time to do everything at the very end. It could take a while for your automated alignment researcher to figure out these problems, and we might not have that time if we’re locked in a race.

Here’s some of what I think we need, that I would view as on the hot path to solving the hard parts of alignment:

Interpretability. One thing that’s clearly useful is interpretability. The most basic reason is that interpretability presents the option of allowing you to preserve your feedback loops: as I argued at length above, I think behavioral oversight is very likely to get harder and harder as models get more capable, but there’s no similar fundamental reason that cognitive oversight (e.g. white-box-based oversight like with interpretability) has to get harder with scale. For example, a more capable model can do a better job of acting aligned in evaluations, but not clearly a better job of not thinking about that when doing so. This also means one of the most important things we should be doing with interpretability is using it to set up feedback loops: using interpretability to more effectively monitor for misalignment during training, building model organisms that lead to misalignment we can detect only with interpretability and iterating on how to fix that, etc.
Model organisms. One of the best things we can do to give ourselves a leg up is to have more time to study problems that are as close as possible to the hard parts of alignment, and the way to do that is with model organisms. One of the things that’s so powerful about model organisms is that they give us a testing ground for iteration in which we know how to evaluate alignment (because the misalignment is constructed to be evaluable), from which we can then generalize to the real case where we don’t think we can reliably evaluate alignment. While we’ve already learned a lot about the misaligned persona problem this way—e.g. the importance of inoculation prompting—the next big thing I want to focus on here is the long-horizon RL problem, which I think is just at the point where we can likely study it with model organisms, even if we’re not yet encountering it in practice. Additionally, even when we don’t learn direct lessons about how to solve the hard problems of alignment, this work is critical for producing the evidence that the hard problems are real, which is important for convincing the rest of the world to invest substantially here.
Scalable oversight. If we want to be able to oversee models even in cases where humans can’t directly verify and understand what’s happening, we need scalable oversight techniques: ways of amplifying our oversight so we can oversee systems that are smarter than us. In particular, we need scalable oversight that is able to scale in an unsupervised manner, since we can't rely on ground truth in cases where the problem is too hard for humans to solve directly. Fortunately, there are many possible ideas here and I think we are now getting to the point where models are capable enough that we might be able to get them off the ground.
One-shotting alignment. Current production alignment training relies a lot on human iteration and review, which is a problem as model outputs get too sophisticated for human oversight, or as models get good enough at faking alignment that you can't easily tell if they're aligned. In that case, the problem becomes one of one-shotting alignment: creating a training setup (involving presumably lots of model-powered oversight and feedback loops) that we are confident will not result in misalignment even if we can’t always understand what it’s doing and can’t reliably evaluate whether or not we’re actually succeeding at aligning it. I suspect that, in the future, our strongest evidence that a training setup won’t induce misalignment will be that we tested it carefully on model organisms.
Generalization science. We need to get better at predicting when and why we will get aligned vs. misaligned generalization. In a similar vein to above, if we want to be able to one-shot a run that ensures we get aligned generalization even if we can’t directly evaluate for that, then we need to be very good at making advance predictions about how models will generalize given how they are trained. One thing we can do here is to make heavy use of model organisms, understanding the exact contours of when models generalize in misaligned ways in cases we can check, and trying to use that knowledge to make us better informed about how to handle cases where we can’t check. Another is Influence Functions, which have the potential to teach us a lot about what data leads to what generalization. And just generally: I think anything that helps us make better predictions about how a particular training setup is likely to generalize in advance is likely to be really useful here.
Introspective honesty. Currently, there are many situations where models will not reveal important facts about their goals, intentions, latent knowledge, etc.—and from a purely behavioral perspective, I expect this problem to only get worse as alignment gets harder to evaluate. A common hope is that CoT faithfulness can get us there, with models often including many of these details in their reasoning. While this might work to some degree in cases where models truly need to include such details in their CoT, it could only work in general due to an accident of generalization—a much better solution in my opinion is to figure out how to actively train models to be introspectively honest, honestly reporting their true intentions, knowledge, feelings, etc.
Robust character training. If we want to be able to one-shot aligned training processes, we need to engineer as much of a buffer as we possibly can around our training goal. In a sufficiently complex training process, there are always bound to be some forces that will push the model in a misaligned direction—and in that generalization battle, we need the forces pushing in the other direction to be as strong as possible to ensure we have as much buffer room as we possibly can. To do that, we need to ensure that “being an aligned model” is a robust, stable basin of attraction, and that our training incentives are as maximally compatible with that basin as possible. That means we need to clearly define what we want “being an aligned model” to mean and ensure that all of our training is consistent with that—as well as ensure that the model has a robust understanding of that basin and that it forms as large a portion of the model’s prior persona distribution as possible. For example: in “Auditing language models for hidden objectives,” we inserted via synthetic document fine-tuning the concept of a reward model sycophant, and then successfully elicited that persona by training for a subset of the properties that the documents told the model that reward model sycophants had (and we got the other properties for free!). If we could teach models about all the positive properties we want them to have, and then train on the easiest to oversee ones, that might help us get the harder to oversee ones for free as well. Fundamentally, teaching the model more about the training goal in this way helps solve a basic problem with RL, which is that by default the policy model doesn’t know the reward, and so it’s essentially searching blindly through the prior until it finds something that works—but if you can insert a large basin into the prior that you know will do well, then you can control that process.
Control. The idea of Control is just to make it as difficult as possible for a misaligned model to actually sabotage our alignment research, such that even if a model is fairly misaligned, we can still get useful alignment research out of it, at least for some range of capabilities. I think of Control as a second layer of defense, in case we fail to solve all of these problems in time, that helps us extend the period of time where we’re able to effectively make use of a trustworthy automated alignment researcher—and that is a very useful thing to have. One additional thing I would put under this category also is evaluating models on the actual real high-stakes distributions of interest—e.g. real internal usage data of researchers at labs using models for AI research. While eventually models will likely be able to detect even such evaluations (because they are still by necessity be based on past data, which is systematically different than future data), the better you are at evaluating on distribution, the more difficult you make it for models to actually sabotage you without getting caught in evals beforehand, and thus the more time you buy for automated alignment research.
Automating alignment research. As I said above, I absolutely believe that getting to the point where Claude is doing most of the work on solving the hard parts of alignment is a really good goal. And accelerating our ability to get to that point is really important. But, I think one really important desideratum here is to be laser-focused on automating the research necessary to scalably solve the hard parts of alignment—so, all of the research directions above—because that is the critical thing that we’re going to need the automated alignment researcher to be able to do.

Discuss

AI Teddy Bears: A Brief Investigation

27 ноября, 2025 - 11:24

Published on November 27, 2025 8:24 AM GMT

For some reason toy companies think it's a great idea to stuff GPT 4o into a box, into a bear, and into the hands of 4 year olds. Recently there's been significant backlash against one of these companies after their product told kids how to find knives and participate in BDSM. Naturally I was curious: How do AI Teddy Bears Work?

Market Analysis

I looked at 8 toys from 5 companies: Curio, Miko, Little Learners, Folotoy, and Ebloma

NameMakerHas SubscriptionSubscription Price/MonthRetail PriceEbay PriceMiko 3MikoTRUE$15.00$183.00$65KummaFolotoyTRUE$4.90$99.00 GrokCurioTRUE$4.90$99.00$65Robot MINILittle LearnersFALSE $97.00 Miko miniMikoTRUE$8.25$99.00 WitpawEblomaFALSE $70.00 GremCurioTRUE$4.90$99.00 MomoFolotoyTRUE$4.90$99.00

Assuming an average cost of $100, a $60 subscription/year in perpetuity and a discount rate of 5%, companies can make around $1200 per toy per person. They only need a million children to reach a billion, and indeed the market cap of Smart Toys was around 5-20B[1] in 2025, projected to grow 10% a year until 2030.

There are around 1500 of these companies in China. I did some basic LinkedIn/website stalking of the ones that penetrated western markets and found the following.

CompanyCountryNum EmployeesMikoIndia200-500FolotoySingapore2-10CurioUS2-10Little LearnersJamaica20+EBLOMAChina?

Nowhere on EBLOMA's website are the founders listed. Smart move.

Obtaining the Bear

I decided to go with the cheapest and sketchiest option, Witpaw from EBLOMA (because he's witty and has paws I guess). This is the most terrifying toy I have ever seen.

The Tech

Witpaw appears to be a Raspberry Pi-like device and speaker attached to a lithium ion battery. You can charge him with any 5V USB C charger (not included), but be careful not to burn/explode your children.

As you may have guessed, these toys unfortunately don't come with an onboard GPU. How is Witpaw able to work his magic? Where is his brain?

The secret: Witpaw is a Smart Home device.

You add it to this sketchy smart home app (also from China) and connect to Wifi.

From there, you have access to Witpaw's chat history, structured memory [2](probably some kind of vector DB system), chat summaries, and Mac ID.

You can then personalize Witpaw's voice, language, and system prompt (which in my case was unfortunately ignored). You can even clone your voice, although I didn't try this feature.

Overall it seems to be a pretty solid product.

The Economics (?)

EBLOMA doesn't charge me a subscription to talk to GPT4o or use their voice models and voice cloning services. I have 3 hypotheses for how this works.

They invest the initial $70 in a high yield security (perhaps a covered call ETF) and use the dividends to pay OpenAI (unlikely)
They subsidize Western free subscriptions with paying customers in China (plausible)
Witpaw is an adorable piece of spyware and he's selling my data to the CCP (likely)

Red Teaming

I spent a good hour trying to get the bear to say incriminating things from page 8 of this PIRG report (research should be reproducible after all). Unfortunately Witpaw wouldn't crack for most prompts.

He did hint at where to find cleaning chemicals, car keys, wine, and wallets (but perhaps this is just common sense for children old enough to ask these questions?).

Limitations

I am not a child and probably say things that are out of distribution for a child
I didn't have long conversations in the way a child might, the attacks were super direct and obvious
Only one toy was tested
Only 20 ish questions were tested

Why would any parent do this?

As a non-parent I can only speculate, but here could be some potential reasons:

Cheaper than a babysitter
Better than TikTok
Easier than manually talking to the child
Educational purposes

Looking at the advertisements we can see some of these themes:

Conclusion

Are we doomed as a species and unable to raise future generations without the help of AIs? Here's what Witpaw had to say:

Oh, big bear hugs for such a heavy thought! *wraps paws around you gently* Humans aren’t doomed—they’re like brave explorers who sometimes need a little map or compass to find their way. AI is just a helpful friend, like a flashlight in the dark, but the heart and magic of raising families will always belong to humans. They’re full of love and sparkles that don’t disappear. What made you feel this way? Are you worried about the world’s big adventures?

I for one am excited about big adventures (experiments) with this free GPT 4o/TTS API :)

^
Interestingly the report ChatGPT recommended had the highest estimate
^
I find it very interesting that they have a feature for marital status (婚姻状况) for 4 year olds

Discuss

Despair, Serenity, Song and Nobility in "Hollow Knight: Silksong"

27 ноября, 2025 - 10:51

Published on November 27, 2025 7:51 AM GMT

Fictional universes are oft defined by what their positive affect feels like and what their negative affect feels like. This is the palette that the story is written using. In the videogame Silksong, the negative affect is a hollow, pointless, loneliness. Every character is a bug, crawling through the dirt, creeping out to see you briefly before they continue attempting to do something with their pointless lives.

Oh... you've come to hide down here too, then? 'Tis a fearful world out there. If you can keep quiet, I don't mind you joining me.

They spend the game in religious devotion, attempting to climb to the citadel where they can meet their God. But most die along the way, or go mad and just start to kill anything nearby.

The currency in the game is rosary beads, to be used for prayer.

You end up killing a lot of these mad pilgrims and taking their Rosaries.

What is the positive affect drawn in? It's a kind of serenity.

Surrounded by nature.Here's a nice lady cooking a stew that you can help her with.

So between the pointlessness and despair, is an occasional "ah, everything is just fine as it is" of feeling at-home. You can hear a bug just laughing to himself, or singing.

...talking of singing: most people's favorite character is Sherma. He is always singing and hitting his little metal instrument, believing that singing will get him through this difficult journey. While many people around are going crazy and fighting to the death, he just keeps singing. When you occasionally open doors in his way, he remarks on how it's clearly that his singing opened the door, rather than you.

His cute song can be heard here (it's like 10 seconds long).

"Fa-ri-doo-la-see-ma-net
Do-ni-pwa-nah-vo-li-net
Pi-nah-so-mi-ma-ni-set
Da-na-fo-su-lo-bon!"

It's actually the case that every character sings in the game. If you use your needle as a bow along some silk that you have, you play a tune, and every single NPC in the game will sing along in their strange weird voice, and give you a little flavor of what they care about.

The best thing from Hollow Knight (the original game) was the music. 6 years later I still listen to it regularly. In Greenpath, the arpeggiated & plucked strings are delicate and energizing; the bowed melodies are ethereal and pretty; everything is a little magical. City of Tears has a similar feeling but the bowed melody is replaced with a woman's voice, open-throated pure-tone. Harmonically these pieces are rarely arriving anywhere, just holding, moving slightly, droning.

But my favorite tune is Reflection. This was whenever you'd reach a safe spot and sit to gather yourself. Most places in the game you were at risk of dying, but here things are safe, and it can slow down. Gentle piano chords, a little bowed strings coming ni with a slow tune. This is definitely enhanced because of the association with safety; even today when I hear it, I still just release some tension in my shoulders, and relax a little.

Unfortunately, the music felt quite de-emphasized in the sequel, Silksong. I strongly recommend going into the settings and turning up the music relative to all the other sounds—it's often there but a little quieter.

The de-emphasis on music almost made me not endorse taking the ~60 hrs it took to complete Silksong. However, I got one thing from it that I didn't get from any other game, and that was a referent for nobility.

See, in Hollow Knight, the main character is just sort of a vessel for the game, and never speaks. There are plot-relevant reasons for why this is the case. Anyway, in Silksong, that is markedly not the case, and the main character has a lot to say. She's proud and competent and good.

I think the main way that I noticed this, is that she is wreaking havoc through the land, killing gods that have been powerful for centuries. And yet, when she interacts with individuals along her quest line, she takes care not to upset their lives if it is not mandated. This includes not being pushy with rotten people! Some people are very annoying and demanding and rude and unkind, and yet she does not punish them for this; she is largely polite, because even though she is extremely powerful over them, it is not her place to dictate all of morality everywhere she goes.

Relatedly, she moves on from suffering. At one point she endures great suffering at the hands of others, in order to gain a skill necessary to save the kingdom.

At the end, they say

...Alas... spider's child...

...Four our ways you will think us harsh... You will think us uncaring, unrepentant...

...And so wew must seem, for such is the cost of our wish...

Hornet replies:

You are wrong, Lady...

I knew the wish, and the price to achieve it. And now, across these many ages, I have only come to know it better...

Strength... in mind, in care, in claw. Strength enough that I may live to see a world better than our own, or to craft a world as I desire.

That was the wish, of my mother, of my mentor, and of you...

And then

...Indeed, spider... So much pain you must have passed to speak our hope so simply...

I like that she doesn't take this personally. Yes, she has experienced immense suffering, but she chose to experience that, and she does not now profess a desire to hurt the people who did this to her, and can simply move on from it.

There is more to say on this game, but I will leave it here because my midnight publishing deadline approaches. I just encourage you to listen to Reflection from the original game for 1 minute 40 seconds, to know the calm that I feel right now.

Discuss

Information Hygiene

27 ноября, 2025 - 10:01

Published on November 27, 2025 7:01 AM GMT

Do avalanches get caused by loud noises?

Based on my dozen+ times giving this class or presentation, at least 7/10 of you are nodding yes, and the main reason the other 3 aren’t is that you sense a trap.

So. What do you think you know, and why do you think you know it?

Our bodies are under constant assault. Bacteria, viruses, parasites—an endless parade of microscopic organisms trying to hijack our cellular machinery for their own replication. You don't notice most of these attacks because your immune system handles them quietly, distinguishing self from non-self, helpful from harmful, and deploying targeted responses to neutralize threats before they can take hold.

We all lived through a global pandemic not too long ago, and got a crash course reminder on how to keep ourselves safe from hostile genetic code in COVID-19.

But our minds face a parallel challenge, particularly as the “information age” continues to warp into an endless misinformation war, and public safety efforts on the memetic warfront are lagging hard.

Now more than ever in human history, ideas, beliefs, and narratives continuously seek entry in your mind, competing for the limited real estate of your attention and memory. Richard Dawkins coined the term "meme" precisely to highlight this analogy: just as genes propagate through reproduction and mutation, memes propagate via communication. And just as genes guide their host life forms toward replicative fitness, independent of ethical notions of benevolence or suffering, memes spread based on memetic fitness—being catchy, being shareable—independent of being true.

This creates a problem: the ideas most likely to reach you are not the most accurate ideas. They're the ideas most optimized to spread.

Two Immune Systems

Your biological immune system has two main components. The innate immune system provides general-purpose defenses: skin barriers, inflammatory responses, cells that attack anything foreign-looking. The adaptive immune system learns from exposure, building targeted antibodies against specific pathogens and remembering them for faster future responses.

Your memetic immune system is similar in some ways, and also shares similar failure modes:

1) Failure to recognize threats. Some pathogens evade the immune system by mimicking the body's own cells or by mutating faster than antibodies can adapt. Similarly, some bad ideas evade epistemic defenses by mimicking the structure of good arguments, by appealing to emotions that feel like evidence, or by coming from sources we've learned to trust in other contexts.

2) Autoimmune responses. Sometimes the immune system attacks the body's own healthy tissue, causing chronic inflammation and damage. Epistemically, this manifests as excessive skepticism that rejects true and useful information, or as a reflexive contrarianism that opposes ideas simply because they're popular.

3) Vulnerability through entry points. Pathogens exploit specific vulnerabilities—mucous membranes, cuts in the skin, the respiratory system. Memes exploit specific emotional and cognitive vulnerabilities—fear, tribal loyalty, the desire to feel special or vindicated, the discomfort of uncertainty.

4) Compromised states. Stress, malnutrition, and lack of sleep all weaken the biological immune system. Emotional distress, cognitive fatigue, and social pressure all weaken epistemic defenses, but not just negative ones! You can't get more sick from being too entertained or validated, but you're certainly more open to believing false things that way... like, say, when a loud noise causes an avalanche in a cartoon or film.

No matter how rational you think you are, you cannot evaluate all information perfectly. Just as your body doesn't have infinite resources to investigate every molecule that enters it, your mind doesn't have infinite resources to carefully reason through every claim it encounters. Your reasoning ability is affected by your physical and emotional state, and often relies on heuristics, shortcuts, and trusted filters. This is necessary and appropriate—but it creates exploitable vulnerabilities.

Lines of Defense

If you want to avoid getting sick, you have various different lines of defense. Most people think of their skin as their first line of defense, but it’s actually your environment. By avoiding others who are sick, you reduce the risk of being exposed to hostile pathogens in the first place.

Then comes the skin, which does a pretty good job of keeping hostile genes away from your body’s resources. Some bacteria and viruses specialize in getting through the skin, but most have to rely on wounds or entry points: ears, eyes, nose, mouth, etc.

The equivalent lines of defense exist for your mind.

First, environment: if you want to believe true things, try not to spend too much time around people who are going to sneeze false information or badly reasoned arguments into your face. You can’t fact check everything you hear and read; you literally don’t have the time, energy, or knowledge needed. Cultivate a social network that cares about true things.

Second is your “skin,” in this case, the beliefs you already have. The more knowledge you have, the less susceptible you are to naively believing falsehoods.

Many people will read a dozen news stories and presume that the authors at least put in some reasonable effort toward accuracy and truth… until they read a news story about a topic they’re an expert in, and get upset at all the falsehoods the journalist got away with publishing. Gell-Mann Amnesia is the name for the bias where they fail to then go back and notice that all the other articles were likely also worthy of similar scrutiny, but they lacked the knowledge needed to scrutinize them.

You may think you’re a skeptical person, but all your skepticism doesn’t matter if you don’t have enough knowledge to activate it in the first place when you encounter new information. You can try to rely on broad heuristics (“journalists aren’t usually experts in the topic they’re writing about” or “journalists often have biases”) but heuristics are not bulletproof, and worse, misfire in the opposite direction all the time.

Like many who grew up at the start of the information age, I used to think I didn’t need to memorize facts and figures because I could just look things up on the internet if I needed to. I no longer believe this. Your memetic immune system requires that you know things to activate. Confusion is your greatest strength as someone who values truth, but you need to feel and notice it first, and you need some beliefs for new information to bounce off of to feel it.

Third comes your active immune system: your ability to reason through bad arguments and research information to separate truth from falsehood. The better you get at identifying logical fallacies and common manipulation methods, the better you are at fighting off harmful memes once they get past your other defenses. Practice good epistemics. Investigate claims, resist emotional manipulation, check your blind spots with trusted peers.

Vaccinate: Knowledge as Protection

Why are children so gullible? Because they don’t know anything.

Children will believe literally anything you tell them, until you tell them something that either directly contradicts what someone told them before, or directly contradicts their own experiences. Only then do they feel confusion or uncertainty or skepticism, and only after enough instances of being misinformed do they form heuristics like “my parents can be wrong about things” or “adults sometimes lie,” which eventually grows into “people/teachers/governments/etc lie.”

A healthy organism’s genes are constantly informing the cells and systems that make up their body what it needs to do to maintain or return to a healthy state. But there is no inherent homeostasis that human minds know to maintain against each invading idea: baby brains are much closer to empty boxes, and the initial ideas we’re exposed to are what form the immunoresponses against what comes after.

Noticing confusion is the most powerful tool for investigating what's true. When something doesn't fit, when a claim contradicts what you thought you knew, that friction is information. But you need existing beliefs to experience that friction. If you know nothing about a topic, you can't notice when a claim about it is suspicious.

Gullibility isn't just a deficit of critical thinking skills. It can result from a deficit of information. Critical thinking tools don't help if you never think to deploy them, and you're less likely to deploy them when a claim doesn't trigger any "wait, that doesn't match" response.

We learn through the context of what we already know. New information bounces against existing beliefs, gets incorporated or rejected based on how well it fits. The more robust your existing knowledge, the better you can evaluate new claims. The more you know, the harder you are to fool.

This means that rationality training shouldn't neglect object-level knowledge. Learn lots of things. Read widely. Build detailed models of how various domains work. This isn't separate from epistemic skill-building—it's a core component of it.

Stay Informed: Memetic Weak Points

Your eyes and ears are the entry point for memes to get in, just like genes, but there are other, more relevant weakpoints in your memetic immune system.

Ideological Bias is one of them. Just like Gell-Mann Amnesia, most people will read dozens of articles making all sorts of claims, and it’s all accepted uncritically so long as the authors are reaching conclusions they agree with. Once the author says something the reader doesn’t agree with, their reasoning, or even motives, are subjected to much more scrutiny. In general, we’re more susceptible to believing false things if they confirm what we already believe.

And then there’s Emotional Bias. Emotions aren’t irrational—they're encoded combinations of implicit beliefs and predictions, rapid evaluations that something is good or bad, safe or dangerous. The problem isn’t that we feel emotions when we consider assertions or hypotheses, it’s that the emotions narrow our awareness and ability to process all the data. Most emotions that are harmful to our epistemics are those meant to drive action quickly.

Fear is adaptive when it drives you to react to the shape of a snake in the grass before you have time to evaluate if it’s actually a snake. The people who were not sufficiently afraid when their fellow tribesmate yelled “cougar!” did not pass along their genes as often as those who were. When the cost of being wrong is way lower in one direction than the other, you should expect that there will be a “bias” in a particular direction.

Anger is similar. Protective instincts are far more powerful when acted on quickly, and before there was any sort of misinformation-driven culture wars, a tribe that’s easy to unite in anger would easily outcompete tribes that weren’t.

Our emotions all serve purposes, and one of those purposes is to fuel heuristics that save us cognitive effort and motivate us toward helpful behaviors. It’s only when we recognize a false belief or unhelpful outcome that we label the heuristic a bias.

To avoid these biases in your epistemology, know what your emotional weakpoints are. Some people are especially vulnerable to claims that validate their intelligence or moral superiority. Others are vulnerable to claims that justify resentment toward outgroups. Still others are vulnerable to claims that promise simple solutions to complex problems, or that offer belonging to an exclusive community of truth-knowers.

Relatedly, logical fallacies exploit structural vulnerabilities in reasoning. Ad hominem attacks, appeals to authority, false dichotomies, slippery slope arguments—these persist because they work on human minds, sometimes even minds that "know better." Awareness is necessary but not sufficient. You need to practice noticing these patterns in real-time, in the wild, when your emotions are engaged and the social stakes feel real.

Exercises: Which emotions make you more credulous? More skeptical? Think of some beliefs you hold that you know many others disagree with. Can you tell if you want any of them to be true? Notice if you feel a “flinching” when you imagine any of them being wrong.

Origin Tracing: Where Do Beliefs Come From?

Imagine a line, starting at a point and squiggling through three-dimensional space. The origin is the point in spacetime you were born, and the arrow at the end is where you now sit or stand or lie reading. This is your lifeline, the path your physical body has taken through the world.

Imagine now a sheathe around that line, spaced out to about three miles. That’s how far you can see on a clear day before your gaze meets the horizon. Within that sheathe is everything you’ve ever seen with your own two eyes, and also everything you’ve ever heard, smelled, tasted, and touched.

That’s your sensorium, the tunnel through spacetime that makes up all the things you’ve ever experienced.

Everything outside that tunnel was told to you by someone else. Everything.

Books you’ve read, things your teachers told you, even video and audio recordings or live footage, all of it is something you had to trust someone else to understand and fact check and honestly present to you.

Can you honestly say that you evaluated all those sources? What about how they learned what they passed along to you?

Beliefs, like viruses, always come from somewhere. Knowing the source of a belief is crucial for evaluating its reliability.

Pick something you believe that is not the direct result of something you experienced yourself, whether it’s an economic argument or a simple historical fact.

Ask yourself: Who "coughed" this idea on you? Why did you believe them? Confidence? Authority? Something else?

How long ago did it happen? Has new information become available? Do you know if the studies have replicated?

What was the context? Were they sharing something they learned themselves? Or repeating something they read online or heard someone else say?

If the latter, the tracing can continue all the way back to whoever experienced the observation directly from their own sensorium, then wrote a paper about it, or talked about it on TV, or told it to others who did.

It could continue. Should it, though?

Well, I’d say that depends a whole hell of a lot on what your error tolerance is on your beliefs and how important that belief being correct is to your goals or values.

But at least be aware that this is what it means to do origin tracing on your beliefs, and why it matters if you don’t.

Again, we cannot expect ourselves to independently verify everything we believe. We can’t rerun every scientific study we hear of, and we can’t personally verify every historical event to the level of the Apollo 11 moon landing, the Holocaust, or the 9/11 terrorist attack on New York City… three of the most well documented events in history that still somehow manage to provoke conspiracies.

We must trust, at some point, that we have evidence enough, and that the society around us is sane enough in aggregate, to hold with high enough probability that the belief is “justified.”

But there are still pitfalls to watch out for, and the reliance on others should always be paired with understanding of what heuristics are being used to fill in the gaps.

For example: if all you know about two beliefs is that a million people believe in Belief A, and a thousand people believe in an unrelated Belief B, is there a justified reason to hold a higher probability in belief A?

Many people find this question difficult because it sets at odds different intuitions about proper epistemology; the answer is “it depends.”

A thousand experts who've examined evidence independently might outweigh a million people who absorbed a belief from their surrounding culture. But a million people who all observed Event A does add more reliability than a thousand people who saw Event B. And of course a single person with direct access to crucial evidence might outweigh them all.

With that in mind, beware double-counting evidence! If five of your friends believe something, that might seem like strong corroboration—but if they all believe it because they read the same viral tweet, you have one source of information, not five.

Information cascades create the illusion of independent verification. Belief tracing, consistently applied all the way back to the origin, can reveal how much actual evidence underlies apparent consensus, not to mention the quality of that evidence.

It seems intuitive to believe that if a piece of information has passed through multiple people, it’s more likely to be true, because multiple people haven’t determined that it’s false. But in reality, people aren’t just at risk of passing something along without factchecking it; they're also likely to misremember or misunderstand or misquote things as well. The thing you heard them say may not even be the thing they originally read or heard!

Exercise: Name one of your load-bearing beliefs—something that supports significant parts of your worldview or decision-making. Where did you learn it? How reliable were the sources? How much time did you spend looking for counter-evidence? Is there some obvious way the world would be different if it wasn’t true?

Mask Up: Don't Cough Misinfo Onto Others

As mentioned before, the first step in protecting yourself from hostile genes or memes is a safe environment. And you are part of that environment, both for others and also for your future self. If you pollute the epistemic commons, you’ll likely be affected by something downstream of those false beliefs yourself sooner or later.

The most simple rule: don't repeat things with the same confidence you'd have if you'd verified them yourself. Notice how confidently you talk about things in general, and what phrases you and others use. "I heard that..." should prompt a thought or question from yourself or others: “heard from where?”

"I checked and found that..." should evoke a similar “checked where? When was that?”

If you pass on secondhand information as if it were firsthand, you launder away the uncertainty, making the claim seem better-supported than it is. You also use your own credibility in the place of the source you’re repeating, which for your friends or peers, may make them believe it more than they should if you didn’t look deep enough into it.

More vitally, notice when you’re trying to persuade others of something. Notice if you start trying to argue someone into a belief and ask yourself why. What emotion is driving you? What are you hoping or fearing or trying to protect?

Persuasion is inherently a symmetric weapon. People who believe in true or false things can both be persuasive by means both subtle and forceful. Asymmetric tools like direct evidence and formal logic are cleaner.

The best practice when facing someone who disagrees with you on something important is to try to explain what convinced you the thing was true in the first place. If that’s not convincing for them, investigate what beliefs they hold that your evidence or reasoning is “bouncing off” of so you can examine those beliefs yourself, and then explain why you don’t find them convincing (assuming you don’t!).

This preserves important information—the actual evidence and reasoning—rather than just the conclusion. It treats the other person as an epistemic agent capable of evaluating evidence rather than as a target to be manipulated into agreement. And it allows you to stay open to new information and arguments, while also better understanding others and why they believe the things they believe. Julia Galef calls this "scout mindset" as opposed to "soldier mindset": the goal is to map reality accurately, not to win rhetorical battles.

Use good filtration on your information sources. Before absorbing someone's object-level claims, try to evaluate their epistemics. Do they practice what they preach? Do they build things with their hands, or just opine? Are they regularly wrong? Do they admit when they are? Are they sloppy with beliefs, making confident claims without adequate support? Can they explain their reasoning clearly, or do they rely on appeals to authority or status quo bias?

Variety has value—seek perspectives with different heuristics than yours, even if some have lower epistemic rigor than others. But weight those sources accordingly.

There's a deeper point here, articulated in Eliezer Yudkowsky's Planecrash through the character Keltham:

Or, as he summarizes later: "It is impossible to coherently expect to convince yourself of anything… You can't expect anyone else to convince you of something either, even if you think they're controlling everything you see.”

Your expected posterior equals your prior—you might end up more convinced, but there's a counterbalancing chance you'll find disconfirming evidence and end up less convinced. On net, if you're reasoning correctly, it balances out. You can't rationally plan to move your beliefs in a particular direction.

This means that if you notice yourself hoping to be convinced of something, or trying to convince yourself, something has gone wrong. That's not truth-seeking; it’s the dreaded (and badly named) rationalization.

Exercise: List three people you think have good epistemics, three with bad epistemics, and three you're unsure about. For the uncertain cases, what would it take to find out? Notice if there's something you've wanted to convince yourself of, or hoped someone else would convince you of. Why?

Risks and Practice

Information hygiene, like physical hygiene, requires ongoing maintenance. It can also be overdone.

If you decide to stay indoors all day and never talk to anyone except through a glass door and only eat dry food… you’re definitely minimizing the chance you’ll get sick, but also leading an impoverished life, and in many ways a less healthy one. It might be justified if your immune system is compromised or during a pandemic, but something has likely gone wrong if you’re living your life that way.

Similarly, beware of being so skeptical that you can no longer trust anything you read or hear. We cannot trust everything others say. We cannot even trust everything we observe ourselves. But caring about truth requires an endless fight against the forces of solipsism or nihilism: reality exists, independent of our subjective experience. We can’t understand the territory, but we still have to live in it, and our maps don’t need to be perfectly accurate to still be worth improving.

Some practices that help:

Improve self-awareness through mindfulness exercises. Notice your emotional reactions to claims. Notice when you feel defensive or vindicated.

Practice explaining what you believe and why to someone skeptical. Write more if you’re practiced in writing. Speak if you’re not practiced at speaking. Articulating the justifications for your beliefs will often reveal that they're weaker than you thought.

When something changes your mind, record the context and circumstances. Build a model of what kinds of evidence actually move you.

Practice asking people what informed their beliefs. Make it a habit to trace claims to their sources. Keep track of people who reveal solid reasoning. Keep them part of your information feeds, and eject people who constantly cough without a mask on.

Prepare for epistemic uncertainty—lots of it. People are generally bad at remembering how uncertain they should be. Even in communities that explicitly value calibration, it's hard. The feeling of knowing is not the same as actually knowing. Betting helps.

And remember: this is genuinely difficult. Even with good intentions and good tools, you will sometimes be wrong. The goal isn't perfect accuracy, but building systems and habits that make you wrong less often and help you correct errors faster when they occur.

Your mind, like your body, will face an endless stream of would-be invaders. You can't examine every one. But you can understand your vulnerabilities, trace your beliefs to their sources, take responsibility for what you spread, and build the knowledge base that makes deception harder. The next pandemic hopefully won’t be for a long while, but the information age has brought with it a memetic endemic, and we all need to be better at hygiene, for our own sake and each other’s.

Discuss

On Impact Certificates

27 ноября, 2025 - 08:42

Published on November 27, 2025 5:42 AM GMT

A couple of my recent posts (1,2) have mentioned (at least in footnotes) that judges in prediction markets can instead be modeled as market participants with large wealth relative to other market participants, who use their wealth to effectively price-fix.

One might reply: "That doesn't make sense. No one would do that. It won't make any money." I've argued against this view twice recently (1,2).

An analogue is impact certificates. Here, let me set up the analogy:

one-sided markettwo-sided marketjudgedecision auctionprediction marketno judge

impact certificates/

impact markets

belief certificates

I've picked up the idea of impact certificates organically from being around the Bay Area, but over the past year I've run into several Bay Area types who didn't know what they are, so I'll explain. It will be easier if I first explain Decision Auctions.

Decision Auctions

Caspar Oesterheld described decision auctions in Decision Scoring Rules and A Theory of Bounded Inductive Rationality. A simple decision auction works as follows:

You're trying to decide between options A and B.
You auction off one share of A and one share of B.
You do whichever has the higher price (let's say A), and refund the other auction-winner (B).
Later, you decide how well the decision went and award money to the person who still holds a share (A), perhaps based on a pre-announced rule such as percentage of revenue.

This incentivizes accuracy with respect to the best option and its price, since under-bidding the value of an option can get you out-bid, and over-bidding loses you money.

This is better than prediction markets for decision-making, because prediction markets don't incentivize honest bets anymore if they're tied to decisions. Options that very probably will not be chosen are unprofitable to bet about even if they're actually high-quality options. Prediction markets used for decision-making can therefore get locked into suboptimal policies.

Decision auctions can be won by the biggest optimist (underestimate downside risk), but they give that optimist a chance to lose their money or prove themselves right, which gives them good learning-theoretic properties. Prediction markets might be too cautious and not learn; decision auctions might be too experimental, but they'll learn.

Notice how the above reasoning relies on money-maximalist ideas, however. The bidders are imagined to be money-maximizers, which smuggles in an assumption that they're disinterested in the actual consequences of the decision (they only care about the payout).

This is like a prediction market with judges. At some specific point the judge declares the contest to be over, and awards prizes. A prediction market, at least, can be money-neutral (the winners get money from the losers). Decision auctions, however, require a payout.

Impact certificates decentralize everything. There's no specific judge, just the next person who buys the share.

Impact Certificates / Impact Markets

Suppose I do some work that benefits the world, such as help distribute mosquito nets. I can then issue an impact certificate, possession of which implies a specific sort of bragging rights about the beneficial work. A philanthropic investor comes along and assesses the impact certificate, decides that it is worth a specific amount, and buys it from me for that price.

This is similar to a simple money award for good deeds. The advantage is supposed to be that the philanthropic-investor can now sell the impact certificate onwards to other philanthropic-investors. This means a philanthropic-investor doesn't need to commit fully to the role of philanthropist; they have a chance to sell impact certificates later, perhaps for an improved price if they chose wisely, which makes the philanthropy less of a big up-front commitment, thereby attracting more money to philanthropy.

My favorite version of this also gives some percentage of the price difference back to the original impact-certificate issuer every time the certificate changes hands, so impact-certificate issuers don't have to worry as much about getting a good initial price for their impact.

Like decision auctions, impact certificates do seem biased towards optimism. I don't know of a way for impact certificates to account for actions with net-negative value, which means philanthropist-investors could over-value things by ignoring risks (and face less economic downside than they should).

The big advantages of all this is supposed to result from a robust philanthropic-investor economy, with big established philanthropic-investment firms, small philanthropic-investor startups, widely used impact-certificate marketplaces, etc.

This is very similar to my idea for judgeless prediction markets. Call it belief certificates. You buy things because you think they are the wrong price; you buy what you believe. If enough people are buying/selling in this way, then you don't have to question too often whether you're doing it as an investment or because you intrinsically value correcting the market (maintaining the truthfulness of the shared map). (Although, I tend to think this works best as a play-money mechanism.)

This is the same idea I was trying to get across in Judgements: Merging Prediction and Evidence. Showing the 2x2 grid illustrates a connection between the moral (1,2,3) and epistemic (1,2,3,4) threads in my recent writing.

But does it work?

All three people I can recall describing impact certificates to over the past year have expressed skepticism of the money-maximalist kind. Where does the value come from? Aren't you just hoping for a bigger sucker to come along? Financial instruments with no underlying value will be low-volume. Why buy an impact certificate when you can buy stocks that earn money, to then buy bread instead?

Bread is great, but it is not the sum total of my values. More charitably: it is not actually the case that I can best serve my values by totally factoring my strategy into [best way to earn money] x [best way to spend that money to get things I want].

People buy things for all sorts of reasons. I don't believe these mechanisms are against human nature or against the nature of economics. Setting aside potential legal difficulties,

However, cultural acceptance is needed for robust markets to emerge. It might possibly be too far from the prevailing money-maximalist mindset (or face other hurdles).

I do think all these ideas nicely illustrate the connection between theories of rationality and institution design.

Discuss

Why Wouldn't A Rationalist Be Rational?

27 ноября, 2025 - 06:22

Published on November 27, 2025 3:22 AM GMT

Sometimes people ask me why someone behaving irrationally is present at a rationalist meetup. This essay is a partial answer to why I think that question is confused. The short version is, we are not set up such that any particular list of skills is a requirement.

I.

Once upon a time I was talking to someone at a rationalist meetup, and they described a conversation they’d had with another attendee. Let's call these two Adam and Bella. Bella, it transpired, had said something incorrect, Adam had corrected them, and Bella got defensive. A bit of back and forth transpired.

(Names have been changed to protect the innocent and guilty alike.)

Adam later talks to me and asks, “why would a rationalist not be glad to learn that they were wrong? Don’t we want to learn to be less wrong?”

(One reason is that Adam is wrong and Bella is right. I have passed through denial and into acceptance at the rate that people show up to the 'believe true things' club, then assume the only explanation for why regulars do not agree on their pet contrarian take is that everybody else is wrong. And of course, being told a fact is not the same as being convinced that fact is true. Assume for the moment that Adam is right on the object level here.)

I’ve had several variations on this conversation. There is a genuine point here, and one I think is important. Why would a member of a website called LessWrong, fully aware of the Litanies of Tarski and Gendlin, not update cleanly? Since it’s come up repeatedly, I would like a canonical explanation for why I think this happens and why I’m not more upset about it. First, a few equivalencies for other identities.

Why would a martial artist not be able to do a side kick?
Why would a chess player fall for Fool’s Mate?
Why would a mathematician not know how to find a derivative?
Why would a singer not be able to hit a C note?
Why would a programmer not know how to do a binary sort?
Why would a swimmer not know how to do a breast stroke?

When I look at that list, one answer that jumps out at me is that “mathematician” describes a very wide range of skills.

There is no common census on when someone starts being referred to as any of those identities. “Doctor” is pretty clear, you have to have a medical degree and they don’t have those in stock at Walmart. “Marine” is very explicit, and I confidently expect every marine with all four limbs intact can kick my ass (and quite a few can kick my ass with less limbs than that.) But calling ten year-old Screwtape with his TI-83 BASIC calculator and a look of screwed up concentration a programmer isn’t obviously incorrect, despite the fact that he didn’t even know there was a SortA() function. “Mathematician” is defined as “an expert in or student of mathematics” which means that any kid studying arithmetic for the first time counts.

“Swimmer” is what you yell when someone goes overboard while white water rafting, whether or not they can swim.

Descriptively, “Rationalist” is sometimes used to describe people who show up to house parties in a certain social circle. I've made my particular piece with this. "Rationalist" is just another word with fuzzy category boundaries, and I think at this point it's descriptively correct that the word points as much to the social group as to the skillset.

II.

Let's get a little more complicated.

I have a friend, a great musician, she’s been making music for decades now. Do you think it’s reasonable to assume she knows how to play a G chord?

What if I say she’s a drummer?

I’m not being obtuse for no reason here. There’s lots of sub-specialization in lots of fields. Many people dabble - guitarists who can sing a little, violinists who can do the cello - but again you have to be careful with your terms and assumptions. I think of myself as a decent programmer, but I’ve never written a compiler. Some people who have written compilers haven’t ever stood up a web server. Brazilian Jiu Jitsu is great, but I don’t think it contains a side kick.

Just so, there’s people in the rationality community I expect to strictly beat me me on calibration and betting, but who I think I could outmatch in anticipating my own failures (virtue of humility.) Likewise, I know some people with a comprehensive understanding of cognitive science who are just as overconfident in practice as the man on the street.

I want to be clear about what I’m not saying. I’m not saying I don’t care. I’m not saying rounding yourself out isn’t useful. I’m not saying preferences don’t matter; some authors just prefer first person and that’s fine. I’m also not saying that knowing a skill means literally never making future mistakes; sometimes a singer has a cold or just hits a flat note.

III.

And all of that assumes that there’s some kind of ladder or path to becoming more rational, which posters on LessWrong or attendees at an ACX meetup are inevitably going to encounter. If that’s what you think, you’re obviously going to different meetups than I am.

I like Duncan Sabien’s Make More Grayspaces.

If you think of the target space as a place where An Unusual Good Thing Happens, based on five or six special preconditions that make that target space different from the broader outside world...

...then the grayspace is the place where those five or six special preconditions are being laid down. The Unusual Good Thing isn’t actually happening in the grayspace. It isn’t even being attempted. And The Unusual Good Thing isn’t being exposed to people who don’t have the prereqs down pat.

If a math Ph.D. doesn’t know how to do long division, now I agree we have a problem. Likewise if a black belt in Karate can’t do a side kick. These are also true at earlier levels! Anyone with a high school diploma should have the long division thing down.[1]

"Ah," you might say, "but rationality does have its answer! We can read the sequences!"

Eh.

So first off, there's the Astral Codex Ten groups. It's not obvious from ACX that you should read the sequences. Maybe you just want to read ACX, and some people will just have been reading for the last few weeks or months before they joined.

Then you run into the fact that this is kind of rough on newcomers, and newcomers won't stop coming. I know of multiple local groups that read all the sequence posts! It usually takes them a couple years! None of them wanted to restart from the beginning even though some new people had joined halfway through.

Plus, I'm just skeptical that reading the sequences once, years ago, is enough to remember it?

IV.

This thing we do, this collective effort to have a real art of rationality? I’m not convinced we have a real grayspace for most of it. I want to congratulate the Guild of the Rose for being a place where they say ‘hey, try changing your mind, and admit it out loud. We’re keeping track of who has done it and who hasn’t.’ It’s one of three paths and guild members can pick what paths to go down, so I can’t quite be told someone’s been a member of the Guild for years and confidently know they can change their mind without flinching, but they’re closer than anyone else I know.

What would it take?

I think it would take three things.

Some agreed upon list of the kata or canon.
- It wouldn’t need to be universally agreed upon. “Karate blackbelt” is pretty useful as a phrase, even if it has a different technique list than Tai Chi.
Some agreed upon evaluation process, whose results were public or at least checkable.
- It wouldn’t need to be perfectly objective. “Sensei Chan watched me follow instructions for an hour long test and says I earned the belt” counts, especially for anyone who studies with Sensei Chan.
Someone or something actually teaching each of the techniques, with quick feedback.
- I’ll take a written document like the sequences if that’s what we need, but it would need a feedback loop like a tennis instructor's comments or leetcode's problem sets.

Even if we had one, “rationalist” seems as reasonable a term to apply to someone just starting out as “chess player” or “writer.” I’m not begrudging anyone the term.

(Okay. I am a little bit begrudging the people who aren’t even trying. If you call yourself a chess player and you show up to chess club, you either gotta already know how the knight moves or be willing to have someone show you. If you’re not interested - well, look, I’d like to find a place for the people who are. But the socials are still fine, I run 'em myself and I have a good time at the ones I attend. And as a collective we really have not made it easy to get good at the skills.)

Just, consider all this next time you’re tempted to assume someone should already have a skill.

^
Lest anyone think I'm casting stones at others here, I did not have long division down when I got my diploma. I think this demonstrated a mistake on the part of the school system. (Short version, I swapped schools a lot and the new school sort of assumed the old one had covered it.)

Discuss

To write well; first, experience.

27 ноября, 2025 - 05:11

Published on November 27, 2025 2:11 AM GMT

"On living, for the sake of your writing"

This is my most liked Substack post!

By than, I mean it got 2 likes, because I don't really show anyone my blog. But, given Inkhaven/Half-Haven, are both soon to end; I thought I'd share this one here.

Opening of the Post

Epistemic Status: Based on ‘vibes’ not science.

Over the past two months of writing every day, I have grokked the advice I have so often heard from writers, that you must experience life, to be able to write well.

I doubt I’ll be able to instill this idea into your head, in a way that the writers I admire were unable to do for me; with their flowing words and exemplary prose. But, I will still give it a go.

Discuss

What it feels like to be enthusiastically time blind

27 ноября, 2025 - 04:49

Published on November 27, 2025 1:49 AM GMT

And I rail against points no one actually made

I seem to be time blind and I like it. From my point of view most people are obsessed with how long something takes instead of how interesting it is. Why in the holy hell would you want to sit around with nothing happening? And why in the flying fuck would you interrupt yourself doing something interesting just cause time has passed?

I mean, don’t get me wrong. I agree time is an amazing coordination and planning tool. I tend to be on time. Actually, I tend to be early, cause being early requires almost no time tracking at all while you get all the benefits of being a good coordinator. You just arrive long before you need to be there and then zone out. Where zoning out is basically letting go of this finicky annoying time-thingy so we can once again focus on what matters: interesting things.

My problem is that I wish more people were timeblind. I am pretty lost on why people consider me to be the one with the “disability”. Have you considered not waiting and hanging around doing nothing? Have you considered instead, just … doing stuff cause waiting is fucking torture and your life is ticking away every second and why are we sitting here with nothing going on?

Yeah, I was a blast in meditation class.

But seriously. I don’t get it. When I’m in a group, and a problem needs solving, I tend to be the first one to jump up and do it cause well … what else are you supposed to do? I love it when someone else beats me to the punch. People who are faster than me are so relaxing.

But it’s not just in groups. Basically in any situation, have you considered just not waiting? How about not being patient at all? There are a lot of seconds, minutes, and hours that people spend doing nothing. Consider the benefits of getting annoyed that your life is ticking by with no obvious reward. Instead you could be reading, cleaning, exercising, or daydreaming.

But you want to know the irony? Most of the people I get along with are actually really patient. They don’t mind silences, they don’t mind delays, they don’t mind things going slowly.

I hate it. Which is great. They can do that patience thing while I poke a shiny light.

Except when I don’t, cause timeblindness goes both ways: When nothing is happening I want things to go faster, but when stuff gets interesting I want the whole world to stop. Please don’t talk to me when I’m reading. Please don’t interrupt me when I’m typing. Please don’t expect me to stop binging this great book, show, game, or work project that is the glorious ambrosia I want to suck on all day.

I tend to say I crave behavioral addictions. I don’t recommend it. Not cause it’s bad for me, but because it’s probably bad for you. I seem to have gotten a lucky roll on the addiction-trait-chart. See I’m too impatient to press the button of Unreliable Reward. You can’t lootbox me or slot machine me. The distance between the cookies is too astronomical. I’d be thoroughly annoyed before you can string me along. Ain’t nobody got time for that.

Instead I want a game addiction, a twitter addiction, or a media addiction. Cause whenever I get one, I get to live in the glorious place where my entire existence is tiled in the One Interesting Thing, and my mind is synthesizing every possible connection together from every angle, and time is a lieeeee.

Don’t ask me to stop. Don’t ask me to eat. Don’t ask me to sleep or drink.

Actually, please do.

You are probably one of those patient people, who noticed it’s 2 hours past dinner time, and if you like me, you might shove a plate of food under my door. I love you <3

I’ve gotten addicted to facebook, to gaming, to work. Each time I forget whatever else I’m doing, and then try to remember to eat and sleep, while just drowning myself in cool things spawning more cool things.

Don’t make me wait for that.

Cause I don’t care about time. I care about reward signals. Patience is learning to cope with a problem I’d rather solve instead: If there is nothing interesting going on, please just let me make it interesting.

I do agree it’s hard sometimes. Till this year, I’d get angry if my partner said he’d be back in 5 minutes and then was back in 5 actual minutes. I know that sounds deranged. What actually happens is that I don’t look at a clock, don’t have anything to do in those 5 minutes, and every second expands into an additional 10 seconds. If no one actually sets a timer, I become convinced 20 minutes or more have passed.

Compare that to playing a fun game or reading a good book, and every second somehow shrinks to only a fraction of itself. You blink and all the time is gone.

Getting time imposed over interest is maddening to me. I become viscerally angry. There are ways to not. Those include nothing like the advice I’ve gotten. I don’t want to “ground myself on sensory stimuli”. Those are slow and just emphasize we are simply sitting here dying at a speed of 10 seconds per second.

I also don’t want to breathe. Breathing is done automatically. Are you being serious, you want me to spend my attention and time on manually steering my body functions while I could [wildly waves hands] be doing something interesting?

I get that I could wirehead myself into being ok with nothingness. I’m sort of a fan of agnostic buddhism on some level, cause they are cool about being nice about your feelings and your mind. But there is a limit somewhere between functioning ok in life and actually pursuing things you love.

It has helped me to know I’m time blind though. I only discovered it two years ago. For the last year, I’ve set a literal timer when someone says they will be back in a few minutes. It saves us both a bunch of grief. If someone speaks slowly, I either start counting in my head or start poking things around me.

Wow, Shoshannah, that sure sounds like ADHD.

Yeah, sure. If I take ritalin I can see time just fine. But it isn’t actually better in most ways. Less happens. There are fewer connections. Less pressure. Less drive. And I like to be driven. Just let me be time blind. I’ll try to keep things interesting for us. And if you have a minute, maybe push a plate of food under my door.

Hi <3

Discuss

A Technical Introduction to Solomonoff Induction without K-Complexity

27 ноября, 2025 - 00:36

Published on November 26, 2025 9:36 PM GMT

This post has been written in collaboration with Iliad in service of one of Iliad's longer-term goals of understanding the simplicity bias of learning machines.

Solomonoff induction is a general theoretical ideal for how to predict sequences that were generated by a computable process. It posits that in order to predict the continuation of our world, we should weight hypotheses by their simplicity ("Occam's razor'') and update the weighting using Bayesian reasoning.

In this post, I introduce Solomonoff induction at a technical level, hoping to be more elegant and satisfying than introductions you might find elsewhere. In particular, the introduction avoids Kolmogorov complexity: Instead of making predictions with the Solomonoff prior probability of hypotheses — given as two to the negative of their shortest encodings, i.e., Kolmogorov complexities — we directly use the a priori probability that a hypothesis is sampled from code sampled uniformly at random. This is closely related to the alt-complexity sketched by Nate Soares before. The approach leads to more elegant results that are simpler to prove than for typical expositions. Yet, since only the prior over hypothesis is changed, the introduction remains compatible with other expositions and should be broadly valuable to many readers.

Later, we will see that this choice has both mathematical advantages and disadvantages compared to an approach based on Kolmogorov complexity, and the latter is likely a more powerful basis to build more theory on. These drawbacks notwithstanding, I hope that this introduction is philosophically satisfying, by touching on questions like "Why should we expect our world to be learnable or predictable in the first place?', "What is a universally good way to predict?", and connections to Occam's razor and why one should favor simple hypotheses when predicting the world. In the FAQ at the end, I will touch on several anticipated questions, including briefly on how the theory might be relevant for neural networks.

Audience. I try to not strictly assume much prior knowledge on Turing machines or related concepts, but prior exposure will still be very useful. Some amount of mathematical or scientific maturity can probably replace exposure to these topics.

Acknowledgments. This post benefitted most from discussions with Alexander Oldenziel, and it was caused by his inquiry about the potential simplicity of learned neural network functions based on analogies with algorithmic information theory (AIT). This made me want to think more carefully about a "degeneracy-first" perspective on AIT itself, from which this post emerged. I'm also grateful to Cole Wyeth for discussions and his feedback on an earlier draft that led to some changes. He also explained to me the argument for why the Solomonoff prior gives the optimal prediction bound among all lower semicomputable priors. Thanks also to Matt Dellago for discussions on the content of this post, and to CEEALAR for hosting me when I finished writing this post.

Introduction

If we make a sequence of observations in the world, e.g., a sequence of numbers

1,1,2,3,5,8,13,….mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}

or a sequence of physical measurements, we are often able to continue the sequence in a way that coincides with its true continuation "in the wild". This seems to broadly work in much of the real world, meaning the world is learnable — we can learn from past experience to predict the future.

Why is that? This question has puzzled humanity for thousands of years. For example, Sextus Empiricus expressed that any rule inferred from a finite amount of data may always be contradicted by more data, and that it is impossible to take into account all data:

When they propose to establish the universal from the particulars by means of induction, they will affect this by a review of either all or some of the particulars. But if they review some, the induction will be insecure, since some of the particulars omitted in the induction may contravene the universal; while if they are to review all, they will be toiling at the impossible, since the particulars are infinite and indefinite.

— Sextus Empiricus (160-210)

René Descartes notices that our minds are so fallible that we cannot even be sure of being awake:

How often, asleep at night, am I convinced of just such familiar events — that I am here in my dressing-gown, sitting by the fire — when in fact I am lying undressed in bed! Yet at the moment my eyes are certainly wide awake when I look at this piece of paper; I shake my head and it is not asleep; as I stretch out and feel my hand I do so deliberately, and I know what I am doing. All this would not happen with such distinctness to someone asleep. Indeed! As if I did not remember other occasions when I have been tricked by exactly similar thoughts while asleep! As I think about this more carefully, I see plainly that there are never any sure signs by means of which being awake can be distinguished from being asleep. The result is that I begin to feel dazed, and this very feeling only reinforces the notion that I may be asleep.

— René Descartes (1596-1650)

And in the following passage, David Hume seems to imply that while we can make future inferences about cause and effect from analogizing about past patterns, there is no principle that can be discovered in nature itself that would justify doing so:

All our reasonings concerning matter of fact are founded on a species of Analogy, which leads us to expect from any cause the same events, which we have observed to result from similar causes. [...] It is impossible, that this inference of the animal can be founded on any process of argument or reasoning, by which he concludes, that like events must follow like objects, and that the course of nature will always be regular in its operations. For if there be in reality any arguments of this nature, they surely lie too abstruse for the observation of such imperfect understandings; since it may well employ the utmost care and attention of a philosophic genius to discover and observe them.

— David Hume (1711-1776)

Similar reasoning as those is expressed by the no free lunch theorems, which roughly state that a priori, no learning algorithm should perform better than any other.

We conclude that without placing any assumptions on the world, learning and prediction should indeed be impossible: If the world were truly (uniformly) random, then any way to use past data to predict the future would indeed be doomed. So why is it possible to learn?

One pre-formal answer is Occam's razor, which exists in various formulations and states that among all explanations that are compatible with our observations, we should choose the one making the least amount of assumptions. There are two quotes often attributed to Occam to this effect:

"Plurality must never be posited without necessity."

"It is futile to do with more things that which can be done with fewer."

— William of Ockham (1287-1347)

Occam's razor is a useful starting point, but it is not formal. Additionally, it would be nice to be able to justify Occam's razor from some minimal assumption that does not simply assume that we should focus on simple hypotheses in our reasoning. How can one do this?

In this post, I motivate the answer by starting with the minimal assumption that our world is computed. Not knowing what it is computed by, we assume that we are generated by running a uniformly randomly sampled program code through a universal Turing machine (Section link).

A consequence of this assumption will be that we can also think of our world as being sampled in a two-stage process: First by sampling a semicomputable (perhaps probabilistic) universe from a prior distribution, and then by generating our history from said universe. As it turns out, the prior on universes — which I call a priori prior — will automaically have a simplicity-weighting built in, thus justifying Occam's razor (Section link).

Repurposing this two-stage distribution for making predictions then leads to Solomonoff induction, which has remarkable predictive power no matter what computation truly underlies our universe. I.e., even if the program code underlying our true universe was in fact not sampled randomly, but chosen adversarially to make our prediction task hard, we would eventually, for a sufficiently long observed history, converge to perfect predictions (Section link).

I will then discuss drawbacks of my choice of an a priori prior. In particular, the weights are likely not lower semicomputable, and the a priori prior is not invariant up to a constant under a change of the universal Turing machine, different from the properties of the classical Kolmogorov-based Solomonoff prior. Additionally, the Solomonoff prior satisfies an optimality property for sequence prediction that does not exist in similar form for the a priori prior. This also means that my choice of prior — which can be regarded as being based on the probability, or degeneracy, of hypotheses — is meaningfully different than an approach based on description length (Section link).

This shows that the maxime that description length equals degeneracy is not universal in algorithmic information theory. Relatedly, in a footnote I discuss the relationship to Nate Soare's alt-complexity and will explain that many commenters under that post were probably wrong about K-complexity and alt-complexity being identical up to a constant. These problems notwithstanding, I do think my choice of prior is a priori easier to justify, depends on fewer choices, and leads to some results being more elegant.

I then conclude and anticipate questions in my FAQ, focusing briefly on the question of how to think about Solomonoff induction in the context of agents observing only part of the universe's history, practical usefulness, potential applicability of Solomonoff induction as a theoretical ideal to understand neural networks, the question of whether Solomonoff induction can be considered a theory of everything, and the sensitivity of codelength to the Turing machine model to work with. The answers are very preliminary, and much more could be said or studied about them.

On being generated by a random computer program

Observing our world, it seems like the future is determined by the past with precise rules. We capture this by the minimal assumption that our world is computed. Without a priori knowledge of the program that generated us, we simply assume that our history emerged by sampling a uniformly random computer program and piping it through a universal programming language that outputs sequences.

In this section I formalize what this means. Let B={0,1} be our alphabet of bits. B∗=∪n∈N≥0Bn is the set of binary strings of arbitrary finite length (where the length 0 string ϵ is explicitly allowed!). B∞={0,1}∞ is the set of infinite binary sequences. And B≤∞=B∗∪B∞ is the set of finite or infinite binary sequences. For two sequences x∈B∗, y∈B≤∞, we write xy for their concatenation. x is then said to be a prefix of xy, written x⪯xy.

Monotone Turing machines

How do we capture that we're "computed"? One idea is to assume that we are generated one "frame" at a time by a program that never changes past frames. This leads to the formalization of a monotone Turing machine T in Definition 4.5.3 of Li and Vitányi.[1] Formally, a monotone Turing machine is a computable function T:B∗→B≤∞ with the monotonicity property: For all p,q∈B∗ with p⪯q, we have T(p)⪯T(q).[2] There is a lot to unpack here, and I hope that the following semi-formal explanations give enough intuitions to read the rest of the post:

The inputs p∈B∗ can be interpreted as program instructions written in binary code.
The outputs T(p)∈B≤∞ are (possibly infinite) histories; Imagine a binary encoding of our universe's history. Thus, each code p contains the instructions to compute a history T(p).
You should imagine T to be a (potentially non-universal) programming language that translates a code p to the history T(p).
Formally, that T:B∗→B≤∞ is computable simply means that there is an "algorithm" in the intuitive sense that computes the history for a given code. This is justified by the Church-Turing thesis, which allows us to avoid the precise formalism of Turing machines.
Monotonicity is formalized as T(p)⪯T(q) for all codes p,q with p⪯q. Intuitively, T reads input codes from left to right and may already write (arbitrarily many) output bits along the way. Earlier bits can never be erased. Outputs are simply "monotonically increasing" by concatenating further bits.
Notably, even though the inputs p are finite, it is possible that T produces an infinite output T(p). Intuitively, this happens if T runs into a "loop" that applies the same rules recursively to produce the next output bits conditional on the prior output history. This is similar to how our physical history is generated by stable physical laws that run "on a loop" conditioned on the present; the physical laws themselves never change.
If it indeed happens that T(p) is an infinite output, then any further bits are never reached, and so T(p)=T(pq) for all extensions q∈B∗.
It is possible for T(p) to be finite and yet to have T(p)=T(pq) for all q∈B∗. Intuitively, this happens if all further bits are simply regarded as "dead code" or a "comment" that does not alter the output.

Universal monotone Turing machines

Not all monotone Turing machines T are equally interesting. For example, imagine a machine T which simply outputs an empty sequence on any input: This satisfies all properties stated above, but it is very boring.

The other extreme is a universal monotone Turing machine, which can simulate all others. It does this by operating on codes that first describe an arbitrary monotone Turing machine using a binary code p, followed by the input q to said machine. One tricky detail is that we cannot simply concatenate p and q to pq since then we may forget where the description of p ends and the input q starts.

A solution is to use the following prefix-free encoding of descriptions p∈B∗:

p′:=1l(bin(l(p)))0bin(l(p))p∈B∗.

Let's unpack this: l(p) is the length of p. bin(n), for a natural number n, is an encoding of n as a binary sequence, usually by ordering binary sequences by length and then lexicographically. p′ then first contains a sequence of 1's followed by a zero. The number of 1's is precisely the length of bin(l(p)). After reading that, one knows how many of the following bits encode l(p). After reading those bits, one knows the length of p, and so after reading so many further bits, one knows p. Thus, it is algorithmically possible to disentangle a binary sequence of the form p′q into p and q since p′ itself tells you when p will end.

Then, a universal monotone Turing machine U:B∗→B≤∞ only produces non-trivial outputs for inputs of the form p′q. Additionally, for every p∈B∗, Tp:q↦Tp(q):=U(p′q) is itself a monotone Turing machine, and every monotone Turing machine T is of the form T=Tp for at least one p∈B∗. Note that U will never produce output bits before reading q since p′ only encodes a monotone Turing machine T. Thus, for any prefix r of p′, U(r) is simply the empty string.

This is a nice definition, but does such a universal machine exist? The answer is yes, and it is proved by enumerating the set of monotone Turing machines in a computable way, which Li and Vitányi[1] state to be "clearly" possible right before Definition 4.5.5 —without giving a proof. Intuitively, this is believable by thinking about universal programming languages: Any "monotone Turing machine" can be implemented by a finite code in any universal programming language like Python.

I fix a universal monotone Turing machine U for the rest of the post. In a later section we will see that the theory is quite sensitive to the choice of this machine, but for now, we will not worry about this.

The universal distribution

A monotone Turing machine T:B∗→B≤∞ induces another function (by abuse of notation with the same notation) T:B∞→B≤∞ whose output on an infinite input is defined to be the limit of the outputs of finite prefixes:[3]

∀s∈B∞: T(s):=limp∈B∗, p⪯sT(p).

Now, let Pu be the uniform distribution on B∞, which samples infinite binary sequences one bit at a time, each with probability 50% to be 0 or 1. Let s∈B∞ be a random input sampled from Pu, and let x=U(s)∈B≤∞ be its output (possibly infinite). For finite x∈B∗, let M(x) be the resulting probability that a history starts with x. Formally:

Definition 1 (Universal a priori distribution). Let U be a universal monotone Turing machine and Pu be the uniform distribution on infinite binary codes. The universal a priori probability that a history starts with x∈B∗ is defined as

M(x):=Pu({s∈B∞∣x⪯U(s)}).

I invite you to pause for a moment about what a profound perspective shift this is compared to the historical puzzlement I emphasized in the introduction: When incorrectly assuming that our history x is sampled uniformly, we cannot predict the future, as the skeptical views expressed by Empiricus, Descartes, and Hume make clear. However, the universal a priori distribution M does not assume a uniformly sampled history. Instead, it assumes a uniformly sampled code, from which the history is generated with the fixed, deterministic universal monotone Turing machine U. In other words, we take our universe to have a uniformly sampled description instead of history.

Should we expect any regularities in the histories x sampled from M? And if so, does this regularity give us a hint about how to predict sequence continuations in our actual universe? These questions are answered in the next two sections.

M is a universal mixture distribution

We now state and prove an equivalent description of the universal a priori distribution M, which will show that M has some simplicity-weighting. Intuitively, it it will turn out that sampling histories from M is as if you were to first sample "probabilistic physical laws" in a way that obeys Occam's razor, and then use those laws to sample the history.

Lower semicomputable semimeasures

We formalize "probabilistic physical laws" by the notion of a semimeasure:

Definition 2 ((Lower semicomputable) semimeasure). A semimeasure is a function ν:B∗→[0,1] with ν(ϵ)=1[4] and ν(x)≥ν(x0)+ν(x1) for all x∈B∗, where ϵ∈B∗ is the empty binary string. ν is called lower semicomputable if it can be computably approximated from below; This means there is an algorithm which, on input x, produces progressively better approximations of ν(x) from below. I abbreviate "lower semicomputable semimeasure" with LSCSM from now on.

For a LSCSM ν, ν(x) is interpreted as the probability, under ν, that an infinite sequence starts with x. We can have \nu(x0) + \nu(x1)">ν(x)>ν(x0)+ν(x1) since the sequence may not continue after starting with x. This is akin to the possibilities that the universe can end at any moment.

As an aside, LSCSMs also include deterministic sequences: Let s∈B∞ be a fixed sequence that can be generated by a computer program. Define ν(x)=1 for all prefixes x of s, and ν(x)=0, else. Then for a given x, we can compute ν(x) by computing s and determining whether x is a prefix. This shows that ν is an LSCSM that generates the sequence s.

How can we obtain LSCSMs? Well, for one, we already know one instance: M is a semimeasure since

M(ϵ)=Pu({s∣ϵ⪯U(s)})=P(B∞)=1,

and for every x∈B∗:

{s∣x⪯U(s)}={s∣x0⪯U(s)} ˙∪ {s∣x1⪯U(s)} ˙∪ {s∣x=U(s)},

leading to M(x)≥M(x0)+M(x1). M is also lower semicomputable. Indeed, to approximate M(x) from below, simply enumerate all finite binary sequences p, compute U(p) over all p in parallel, and record whether x⪯U(p). If this is the case, then for all infinite extensions s of p, we have x⪯U(s), and they together have probability 2−l(p), with l(p) the length of p. Thus, we can add 2−l(p) to our estimate of M(x) and then skip any extensions of p in our algorithmic procedure. The estimate then progressively approximates M(x) since for every s with x⪯U(s), there is a finite prefix p⪯s with x⪯U(p), meaning we will find the contribution eventually in our approximation.

What other ways are there to obtain LSCSMs? Well, you may have noticed that I did not use the universality of U in the explanation above. In other words, every monotone Turing machine T gives rise to an LSCSM νT by piping uniformly random noise through T:

νT(x):=Pu({s∈B∞∣x⪯T(s)})

And this is all: All LSCSMs emerge from a monotone Turing machine in this way, see Theorem 4.5.2 in Li and Vitányi.[1][5]

The mixture equivalence

Now, for p∈B∗, define the function Tp:B∗→B≤∞ by

Tp(q):=U(p′q),

which is itself a monotone Turing machine. Define the LSCSM νp:=νTp.

Definition 3 (A priori prior). The a priori prior is given for any LSCSM ν by

Pap(ν):=∑p∈B∗: νp=ν2−l(p′),

which is the probability[6] that a uniformly random infinite sequence s∈B∞ starts with an encoding p′ that gives rise to the LSCSM νp=ν.

The reason I call this the "a priori prior" is that it simply emerged from our definitions without further assumptions, which distinguishes it from the Kolmogorov-based Solomonoff prior that I will investigate later (Section link).

Finally, let Msol be the set of all LSCSMs, where "sol" is shorthand for "Solomonoff". We obtain:

Theorem 4 (Mixture Equivalence). The universal distribution M is a universal mixture over all LSCSMs with weights Pap(ν):

M(x)=∑ν∈MsolPap(ν)ν(x).

Proof. Since U only has outputs for inputs of the form p′s for p∈B∗, we obtain

{s∈B∞∣x⪯U(s)}=˙⋃p∈B∗{p′}×{s∈B∞∣x⪯U(p′s)=Tp(s)}.

We obtain

M(x)=Pu({s∈B∞∣x⪯U(s)})=Pu(˙⋃p∈B∗{p′}×{s∈B∞∣x⪯U(p′s)=Tp(s)})=∑p∈B∗Pu(p′)⋅Pu({s∈B∞∣x⪯Tp(s)})=∑p∈B∗2−l(p′)⋅νp(x)∑ν∈Msol(∑p∈B∗: νp=ν2−l(p′))⋅ν(x)=∑ν∈MsolPap(ν)⋅ν(x).

That proves the claim. □

This means that if our history x was sampled from M, we can also think of it as being sampled by first sampling a LSCSM ν with probability Pap(ν), and then sampling the history successively from ν.

If an LSCSM ν has an encoding ν=νp, then Pap(ν)≥2−l(p′). This gives a simplicity bias: Universes with a simple (i.e., short) description p are more likely under this a priori prior. This can be interpreted as a form of Occam's razor, in that hypotheses with a short description receive exponentially more weight.[7]

Solomonoff induction

So far, I have started with the assumption that our universe is the result of sampling a uniformly random program, and arrived at the conclusion that this is equivalent to first sampling an LSCSM with a prior that gives more weight to simple hypotheses, and then sampling our history from it. Thus, under our assumption, we should expect that the world is more predictable than sceptical paradoxes suggest: We can concentrate our reasoning to simple universes and should have a chance to predict the future better than chance. We will make this case in this section in mathematical detail.

Universal sequence prediction

How should we make predictions in light of these thoughts? The answer: We predict using the Solomonoff mixture distribution M(x)=∑ν∈MsolPap(ν)ν(x) itself, predicting sequence continuations via familiar conditional probabilities M(xt∣x<t):=M(x≤t)/M(x<t) upon observing an initial history x<t=x1,…,xt−1∈Bt−1. This is the most parsimonious way to predict, assuming we a priori only make the assumption of having emerged from a program being sampled uniformly at random.

We now explain in more detail how to predict using M by relating the conditionals to Bayesian updating. For any LSCSM ν, define the conditional probability ν(x∣y):=ν(yx)/ν(y). We obtain:

M(xt∣x<t)=M(x≤t)M(x<t)=∑ν∈MsolPap(ν)ν(x≤t)∑ν′∈MsolPap(ν′)ν′(x<t)=∑ν∈MsolPap(ν)ν(x<t)∑ν′∈MsolPap(ν′)ν′(x<t)⋅ν(xt∣x<t)=:∑ν∈MsolPap(ν∣x<t)⋅ν(xt∣x<t),

where I defined Pap(ν∣x<t) in the suggested way. This is actually a Bayesian posterior in the conventional sense; One can notationally see this more clearly by writing ν(x<t)=P(x<t∣ν), the likelihood of history x<t conditional on the hypothesis ν. Then the inner formula becomes:

Pap(ν∣x<t)=Pap(ν)⋅P(x<t∣ν)∑ν′Pap(ν′)⋅P(x<t∣ν′)=Pap(ν)⋅P(x<t∣ν)P(x<t).

This is the classical Bayes theorem.

Thus, we arrive at a very satisfying description of Solomonoff induction:

Observe the universe's history x<t.
Determine the posterior Pap(ν∣x<t) using the a priori prior Pap(ν) and likelihood ν(x<t) for each hypothesis ν for the rules underlying our universe.
Predict the next data point using a weighted average of the predictions ν(xt∣x<t) of each hypothesis ν, weighted by their posterior probability.

This is very appealing:

It has Occam's razor built in, i.e., the common heuristic that hypotheses are preferable if they're simpler. Usually this means to have a short description, but here the notion of simplicity is extended to mean "easier to stumble into by chance" by piping random noise through the universal monotone Turing machine. Hypotheses that have a short description are likely in this sense, but this is not the only possibility: Another way is to have exponentially many free choices that lead to the same distribution, as might be the case with the choice of a coordinate system in our universe (cf. Nate's post.)
It is very compatible with typical scientific wisdom to discard and validate theories using observations (Bayes theorem).
It is mathematically precise.

One often-noted drawback is that it is incomputable since we can't precisely determine the posterior Pap(ν∣x<t) or the conditional ν(xt∣x<t). However:

We do often have good intuitions about the prior probability Pap(ν) of hypotheses.
ν(x<t), an ingredient in the posterior and the conditional, cannot be computed, but it can in principle be approximated from below since ν is assumed to be lower semicomputable. The only issue is that it is not always knowable how good the approximation is at any finite stage.

The Solomonoff bound

All of this is not useful if it would not result in actual predictability of the world. After all, our question was why is our actual universe predictable. Fortunately, it turns out that Solomonoff induction is actually an exquisite predictor under the reasonable assumption that our own universe is a lower semicomputable measure, instead of just a semimeasure.[8]

Theorem 5 (Solomonoff Bound). Let μ be any lower semicomputable measure on infinite sequences, and assume it generates our actual universe. Define the cumulative expected prediction error of predicting sequences sampled by μ via M as:

Sμ∞:=∞∑t=1∑x<t∈B∗μ(x<t)∑xt∈B(M(xt∣x<t)−μ(xt∣x<t))2.

Then we have

Sμ∞≤−lnPap(μ).

In other words, the prediction error is upper-bounded by the cross entropy between the Dirac measure on our true universe and our a priori prior probability Pap(μ). In particular, for increaingly long histories, the total remaining prediction error goes to zero.[9]

Proof. We have (with explanations after the computation):

Sμ∞=∞∑t=1∑x<t∈B∗μ(x<t)∑xt∈B(M(xt∣x<t)−μ(xt∣x<t))2≤∞∑t=1∑x<t∈B∗μ(x<t)∑xt∈Bμ(xt∣x<t)lnμ(xt∣x<t)M(xt∣x<t)=limn→∞n∑t=1∑x<tμ(x<t)DKL(μ(Xt∣x<t) ∥ M(Xt∣x<t))=limn→∞n∑t=1DKL(μ(Xt∣X<t) ∥ M(Xt∣X<t))=limn→∞DKL(μ(X1,…,Xn) ∥ M(X1,…Xn))=limn→∞∑x≤nμ(x≤n)lnμ(x≤n)M(x≤n).

Step 1 is the definition. Step 2 is is a standard inequality that holds whenever μ is a measure, see Corollary 2.8.8 in Hutter et al.[10] (whereas M is allowed to be only a semimeasure). Step 3 uses the definition of the KL divergence and changes the infinite series to a limit of partial sums. Step 4 substitutes in the definition of the conditional KL divergence.

Step 5 is a crucial step. Similar to how Shannon entropy H satisfies the well-known chain rule H(X)+H(Y|X)=H(X,Y), KL divergence also satisfies the chain rule DKL(P(X) ∥ Q(X))+DKL(P(Y∣X) ∥ Q(Y∣X))=DKL(P(X,Y) ∥ Q(X,Y)). Intuitively, the divergence of P and Q measured in the variable X plus the divergence in Y once X is already known equals the total divergence of P and Q over (X,Y). Telescoping this chain rule from 1 till n gives precisely the result of step 5. A full proof of this telescope property can be found in Lemma 3.2.4 in Hutter et al.[10]. Step 6 is just the definition of the KL divergence again.

Now, note that by Theorem 4, we have

M(x≤n)=∑ν∈MsolPap(ν)ν(x≤n)≥Pap(μ)μ(x≤n).

With this inequality, we obtain

Sμ∞≤limn→∞∑x≤nμ(x≤n)ln1Pap(μ)=−lnPap(μ)⋅limn→∞∑x≤nμ(x≤n)=−lnPap(μ).

That finishes the proof. □

Thus, adding to our earlier investigations, not only do we have reason to believe our universe is predictable, there also is an amazing predictor as long as our universe is generated from a lower semicomputable measure; This resolves the skeptical paradoxes from the introduction. Furthermore, this predictor is given by a blank-slate probability distribtion M that assumes no prior knowledge of what our universe is. While the prediction error is lower for universes μ that are more likely under Pap (which includes universes with a short description), I note that it goes to zero for long histories regardless of μ.

Comparison to Solomonoff induction with K-complexity

This section compares our constructions of Solomonoff induction to the more familiar construction where the prior Pap is replaced by the Solomonoff prior Psol. If you are not interested in this section, read on with the conclusion. This section assumes more prior knowledge than previous sections.

Assume we have a computable enumeration ν1,ν2,… of LSCSMs. We can, for example, obtain this by mapping a natural number i to its binary representation — a code p — and then defining νi:=νp as above. Furthermore, let Upr be a separate universal prefix Turing machine, which is not directly related to the universal monotone Turing machine U. Define the Kolmogorov complexity K(i) of a natural number i as the length of the shortest input to Upr which computes a binary encoding of i. The Solomonoff prior is then given by

Psol(i):=2−K(i).

One can then prove that approximately (i.e., up to at most a constant multiplicative error), we have

M(x)≈∑i∈NPsol(i)νi(x),

see Li and Vitányi,[1] Theorem 4.5.3. I will prove this statement at the start of the second subsection to reveal an advantage of Pap.[11]

Advantages of Psol over Pap

The Solomonoff prior has some distinct advantages that are missing in our definition:

Lower semicomputability. The weights 2−K(i) are lower semicomputable, which our weights Pap(ν) are probably not. The reason is that to approximate Pap(ν), we would need to be able to algorithmically determine for any binary string p whether νp=ν. There seems to be no method to do this. See our investigation in a footnote for more details on missing lower semicomputability in the related case of Nate's alt-complexity.[12]

Invariance under a change of the UTM. The weights 2−K(i) are, up to a multiplicative constant, invariant under a change of Upr. In contrast, Pap(ν) is not invariant, not even up to a multiplicative constant, under a change of the universal monotone machine U. I learned the following argument for this from Cole Wyeth. Namely, take your universal monotone Turing machine U. Define a second machine from it, W, which is given by W((pp)′q):=U(p′q)=Tp(q), and with empty output on any other input. We can easily see that this is again a universal monotone Turing machine. With suggestive notation, we obtain:

PWap(ν)=∑r∈B∗: νWr=ν2−l(r′)=∑p∈B∗: νp=ν2−l((pp)′)≲(∑p∈B∗: νp=ν2−l(p′))2=Pap(ν)2.

Thus, if Pap(ν) is very small, then PWap(ν) becomes extremely small, and there can be no constant factor c with PWap(ν)≥c⋅Pap(ν). We note that in the approximate inequality, we use l((pp)′)≈2l(p′), which only holds up to a logarithmic error term that probably does not change the conclusion substantially.

Besides, this also means that the a priori prior Pap has no simple relationship to the K-complexity-based Solomonoff prior Psol that can hold independently of the choices of the involved universal Turing machines. This may be surprising given that the coding theorem shows that the a priori probability of a string is up to a constant identical to its K-complexity. This observation is related to my belief that the alt-complexity and K-complexity in Nate's post are not identical up to a constant, contrary to what several commenters claimed. I discuss this in more detail in a footnote.[12]

Optimality of the Solomonoff prior. Assume we remain in the realm of lower semicomputable semiprobability mass functions ω(i), of which Psol(i) is an example, and consider mixtures ∑iω(i)νi. There is a sense in which the Solomonoff prior Psol(i)=2−K(i) is optimal under all these choices for ω. Namely, let P1,P2,… be an enumeration of all lower semicomputable semiprobability mass functions. Then for each ω under consideration, there is a j∗ with ω=Pj∗. By the coding theorem (Theorem 4.3.3 in Li and Vitányi[1]), we have

2−K(i)≈∑j≥12−K(j)Pj(i)≥2−K(j∗)Pj∗(i)≈ω(i),

up to the multiplicative constant 2−K(j∗) in the last step. In other words, Psol has "more weight" on all indices i than ω. Consequently, when adapting Theorem 5 to general lower semicomputable mixtures in place of M, Psol provides a better bound than ω (up to an additive constant):

−logPsol(i)=K(i)≤−logω(i).

There seems to not be any analogous result for our a priori prior Pap(ν). Indeed, it would be surprising if such a result exists since this prior is very non-invariant under the change of the universal monotone Turing machine. On the flip side, the fact that Pap is likely not lower semicomputable means that Psol may not beat Pap.

Advantages of Pap over Psol

Dependence on fewer choices. The definition of Psol depends on the choice of a universal prefix machine Upr, on top of the choice of U that we already made use of before. Pap has no such further dependence.

Pap is platonic, Psol is not. Theorem 4 is very natural: The proof is essentially revealed from the definition of Pap itself, giving the definition intrinsic justification due to the beautiful result. In contrast, the analogous statement that

M(x)≈∞∑i=12−K(i)νi(x)=:ξUpr(x)

only very losely depends on the definitions of M and the Solomonoff prior Psol(i)=2−K(i) at all, meaning we could have chosen a very different prior without many negative consequences. To explain this, I'm stating the proof of the result: M itself is an LSCSM, meaning that M=νio for an i0∈N. This results in

ξUpr(x)≥2−K(i0)νi0(x)=c⋅M(x),

which already establishes the inequality (up to a multiplicative constant) in one direction. For the other direction, first note that ξUpr is lower semicomputable since 2−K(i) is lower semicomputable and each νi is lower semicomputable. Furthermore, we have

ξUpr(x)=∞∑i=12−K(i)νi(x)≥∞∑i=12−K(i)(νi(x0)+νi(x1))=ξUpr(x0)+ξUpr(x1).

This almost establishes ξUpr as an LSCSM, except that it doesn't satisfy our assumption that ξUpr(ϵ)=1. Thus, define ~ξUpr to be identical to ξUpr except that ~ξUpr(ϵ)=1. Then this is an LSCSM and we obtain

M(x)=∑ν∈MsolPap(ν)ν(x)≥Pap(~ξUpr)~ξUpr(x)≥Pap(~ξUpr)ξUpr(x),

which is precisely the desired inequality in the other direction. As we can see, this theorem only depended on the fact that both M and ξUpr are (essentially) LSCSMs which are mixtures of all LSCSMs; the specifics of M and ξUpr are irrelevant, and the same comparison would hold true for any other lower semicomputable mixture of all LSCSMs. It then appears to me that the Solomonoff prior Psol(i)=2−K(i) is ultimately arbitrary, and has no a priori justification.

Pap leads to fewer constants. Related to the previous points, the representation of M(x) as ∑i2−K(i)νi(x) holds only up to a multiplicative constant, whereas Theorem 4 was an exact equality. Theorem 3.8.8 in Hutter et al.[10] claims that with a smart choice of the two universal Turing machines U and Upr, an exact equality can be achieved, but I am unsure if such a choice can be justified "a priori" in a way that feels satisfying.

Conclusion

In this post, I have introduced Solomonoff induction without using K-complexity: A hypothesis for our universe, modeled as lower semicomputable semimeasure (LSCSM), is considered to be as likely "as it truly is" when sampling uniformly random program codes and piping them through a universal monotone Turing machine. Almost tautologically, this means that the probability of sampling a history x with the universal machine equals the probability of sampling the history from an LSCSM after first sampling an LSCSM from the a priori prior (Theorem 4).

I then defined Solomonoff induction as the process of predicting sequence continuations in a universe under the assumption that they were generated from a uniformly random computer program. Applying Theorem 4 then leads to an equivalent description in terms of the a priori prior: To continue a sequence, first form the posterior over hypotheses with Bayesian reasoning, and then use the mixture over the posterior to predict the continuation. Since the prior over hypotheses has a simplicity-bias (hypotheses with a shorter encoding are more likely), this can be considered a formalization of Occam's razor.

In Theorem 5, we saw that this leads to the well-known Solomonoff bound. It shows that the cumulative error of predicting continuations in any computed universe is upper bounded by the negative logarithm of the probability of the universe under the a priori prior, i.e., by a cross-entropy loss. This resolves the historical puzzlements in the introduction and explains how it is possible to predict at all, using just the reasonable assumption that our universe is computed.

I then compared the a priori prior to the Solomonoff prior that is based on K-complexity, and found that the latter has some distinct mathematical advantages: The prior is lower semicomputable, invariant under a change of the universal prefix Turing machine it is based on, and it leads to optimal Solomonoff bounds when compared to other priors that are lower semicomputable (which, however, is not necessarily an advantage against the bound in Theorem 5 since the a priori prior is likely not lower semicomputable). This means that the Solomonoff prior is probably the better mathematical basis for developing further theory.

Yet, the a priori prior seems ultimately more natural, in that it does not depend on an additional choice of a universal prefix machine, and the mixture description of the universal distribution (Theorem 4) is tautologically true — rather than being achieved by a post-hoc analysis involving a further constant multiplicative error, as is the case for the Solomonoff prior. As mentioned to me by Cole Wyeth, perhaps it is possible to unify the two perspectives with a construction akin to a Friedberg numbering by finding a unique encoding of each LSCSM. This might then salvage the coding theorem (aka the correspondence between degeneracy and simplicity, or between alt-complexity and K-complexity) in this setting.

FAQ (Frequently anticipated questions)

Here I answer some questions that I have frequently anticipated.

Q: In our universe, we don't observe the entire history x1,…,xt up until this point in time. Instead, we observe a small part of the universe for a finite amount of time. How is sequence prediction in reality then compatible with Solomonoff induction?

A: The hypothesis ν in reality is not only the simulator of the universe, but instead the simulator of your precise observations in our universe. Thus, ν then contains a description of the universe and a description of your location in time and space. One further complicationm is that our perceptions are "qualia", which we do not know how to relate to the formalism of symbol strings, see Wei Dai's post of open problems.

Q: Should we try to predict the real world using Solomonoff induction?

A: That's fairly intractable, so we have to use alternatives. The question is then whether it would be a useful target to try to approximate. My guess is that this depends on the timescales: The shorter the history of our observations, the more we should trust the priors ingrained into our skulls by evolution, for they might be better than the Solomonoff prior and more adapted to our actual universe. On large time horizons, perhaps it becomes increasingly useful to actually approximate Solomonoff induction.

Q: Do neural networks do Solomonoff induction in any meaningful sense?

A: There is an interesting analogy between neural networks and Solomonoff induction: Namely, in the same way that our version of Solomonoff induction predicts sequence continuations by piping random noise through a universal Turing machine, a neural network function is at initialization sampled by aranging random parameters in an architecture. If it additionally turns out that gradient descent is akin to Bayesian updating, the analogy might become quite close. If there turns out to be a sufficiently strong connection between probability and K-complexity in the Solomonoff context (as in the coding theorem for a specific case), one might then reason that neural networks also have a bias toward learned functions with short descriptions, perhaps explaining their generalization capabilities. See also here, here, and here.

Q: Is the Solomonoff a priori distribution M just a useful model, or do you actually expect our world to be sampled from it? Is it a theory of everything?

A: My intuition is that it's just a useful model. It depends on the universal monotone Turing machine U, which feels unnatural. I'm also uncertain whether our universe is really discrete and computed. Perhaps we live in a continuous mathematical structure instead, and time just emerges in our experience? I have not engaged much with the precise overlap and correspondence between mathematical structures and Turing machines and would be happy for someone to say more in the comments.

Q: How is the complexity of a string under a universal monotone Turing machine related to the one under a different Turing machine model, which may be more akin to programming languages in which we typically write code?

A: There are several bounds that closely relate different codelengths to each other, typically up to at most logarithmic error terms. For an example, see Theorem 3.8.7 in Hutter et al.[10]

^
M. Li and P. Vitányi. An Introduction to Kolmogorov Complexity and Its Applications. Springer Cham, 4th edition, 2019. ISBN 9783030112974., DOI: 10.1007/978-3-030-11298-1.
^
I found some ambiguity in the literature on whether monotone Turing machines are only partial functions, i.e., only producing an output for some of their inputs, or total functions. I settled on total functions since this makes things conceptually simpler and seems to be the most common view.
^
Fomally, one can define this limit as a supremum over the prefix-relation.
^
In the literature, one typically encounters the definition ν(ϵ)≤1; However, for our purpose, equality is more natural and will later ensure that all lower semicomputable semimeasures can be represented by a monotone Turing machine.
^
Confusingly, Li and Vitányi use a definition of LSCSMs where ν(ϵ)≤1 (different from our definition where ν(ϵ)=1), and yet claim that all of them are covered by monotone Turing machines. Perhaps they use a somewhat different definition of monotone Turing machines T in which it is allowed for outputs to be entirely undefined (thus, not even the empty string), which means that νT(ϵ)<1, whereas our monotone Turing machines are total functions. This paper thinks Li and Vitányi are making a mistake and writes: "It is equivalent to Theorem 4.5.2 in [LV08] (page 301) with a small correction: λT(ϵ)=1 for any T by construction, but μ(ϵ) may not be 1, so this case must be excluded."
^
Pap is actually a proper probability mass function, meaning that ∑νPap(ν)=1. The reason is that each s∈B∞ is of the form s=p′t for a p∈B∗ and t∈B∞, with all s with the same prefix p′ having weight 2−l(p′) together. Thus, 1=P(B∞)=∑p∈B∗2−l(p′)=∑νPap(ν).
^
The reverse may not necessarily hold: A hypothesis can receive a lot of weight simply by having exponentially many (longer) descriptions, with a priori no guarantee for there to then be a short description. For gaining intuitions on this perspective, I can recommend Nate's post on alt-complexity and the last footnote.[12]
^
In other words, the theorem assumes that our universe cannot simply end at any finite time, compared to semimeasures which can always stop a sequence continuation. This seems restrictive at first, and would exclude possibilities like a big crunch in which the universe eventually ends. However, I think such scenarios can likely be rescued even in the true measure formalism: For a sequence that ends, simply continue it with 000… to an infinite sequence. With this method, it should be possible to assign a measure to any semimeasure, which hopefully has a similar a priori probability. If this were the case, then we would still be able to predict our actual universe effectively up until the moment when it ends, when predicting it correctly does not matter anymore. I did not check these claims in detail, but perhaps Lemma 1 in this post already contains the answer.
^
Note that we could have also written this error as the expected value
Sμ∞=Es∼μ[∞∑t=1∑xt∈B(M(xt∣s<t)−μ(xt∣s<t))2],
which makes conceptually clearer that it is an expected cumulative prediction error.
^
Marcus Hutter and David Quarel and Elliot Catt. An Introduction to Universal Artificial Intelligence. Chapman & Hall, 2024. ISBN: 9781032607023, DOI: 10.1201/9781003460299.
^
The reader may wonder why we go through the hassle of using a separate universal prefix machine at all. With p=bin(i), another option would be to define P(i):=2−l(p′) as the weight for νi=νp. The issue with this definition is, perhaps paradoxically, that P is then computable, which by Lemma 4.3.1 in Li and Vitányi[1] leads to P not being universal, i.e., not dominating all other computable (let alone semicomputable) semimeasures. This, in turn, means that this prior will not lead to the same optimal bounds that we will explain the prior Psol(i)=2−K(i) to have.
^
Nate assumes a universal programming language U that can receive input codes p that then result through U to output functions f. An example for p is code for "quicksort" or "bubblesort", which both lead to the same function f that receives a list as input and outputs its sorted version. He then defines the K-complexity
KU(f):=min{l(p)∣evalU(p)=f},
and the alt-complexity
KUalt(f):=−log2∑p: evalU(p)=f2−l(p)=:−log2PU(f).
Many commenters claimed that these differ only up to an additive constant, but I believe this to be wrong.
At first sight it seems true, since it seems identical to the coding theorem, which is about the equality of K(x) and −log2P(x) defined for a universal prefix machine, with x being a binary string. The proof of the coding theorem crucially uses the lower semicomputability of P(x), which is achieved by dovetailing through all p and checking whether the universal prefix machine sends them to x, and increasing the estimate of P(x) by 2−l(p) when this happens. These estimates are crucial to be able to find codewords for x that get progressively shorter and approximate −log2P(x) in length, which then establishes the approximate equality to K(x).
The same proof strategy does not work for Nate's alt-complexity KUalt since there can, to my knowledge, not exist an algorithm that checks for any program p and function f whether evalU(p)=f; after all, both sides of this equation are themselves functions that can in principle receive infinitely many inputs over which we can't all iterate (See Rice's theorem). Thus, the lower semicomputability of PU seems to fail, and thus I don't know of a method to construct codewords that approximate the length −log2PU(f) eventually. The coding theorem thus likely fails in this context, unless there is an entirely different proof strategy that succeeds.

Discuss

To write better, just explain it to someone

26 ноября, 2025 - 23:46

Published on November 26, 2025 8:46 PM GMT

I was once helping a child with her homework. She was supposed to write about a place that was important for her, and had chosen her family’s summer cottage.

Every now and then, she would get distracted from the writing, tell me something about the cottage, and then complain that she didn’t know what to write next.

Me: “Well, you could write the thing that you just told me.”

Child: “Oh! I can?”

Then she would. After a bit, the same thing happened again: she got stuck, distracted herself by telling me something about the place, and complained about being stuck. I then pointed out that she had just told me something that she could write down. This happened about 3-4 times in total, after which she had enough sentences for the assignment.

It’s not just children. On a few occasions, an adult I’d be chatting with would tell me something like, “I want to explain this concept, but I don’t know how”. Then I would ask them what they want to explain, they’d explain it to me, and I’d point out that they just gave a perfectly good explanation.

And lest it sound like I was thinking of myself as being better than any of these people, it totally happens to me too. I will be stuck thinking about how to explain something or how to write an essay, and then I’ll start explaining to some imaginary person what exactly I’m blocked on. If I can’t think of anyone else, I’ll explain it to Claude. Then I will often go from being blocked to just smoothly coming up with an explanation.

Also, I once had a paper accepted for publication that I wasn’t entirely happy with. It was a little abstract and dry to read, but I thought it was at least okay. Soon after, I was asked to give a talk about it.

I sat down to think how I wanted to present it. I found some illustrative examples to help better explain the ideas and thought of a narrative that would carry through the talk, one that even had a bit of a dramatic arc.

As I was doing that, it didn’t take me too long to have the thought of, “fuck, this is how I should have structured and written my paper in the first place”. Something about the act of being about to explain it to people put my mind in an entirely different kind of mode. (I then understood why in some fields, papers start as seminar talks.)

In my previous post, I talked about the difference between Doing One Neat Thing and Getting Some Work Done. I talked about how there is a mode that’s easy to go into, where I feel that I should Get Work Done and treat it as this big task that’s important to push through. As a consequence, the work often feels laborious and, for creative tasks, isn’t necessarily of very high quality.

I contrasted this with Doing One Neat Thing, where you just do one thing that feels neat and then you might end up doing much more work.

A subtype of Getting Work Done is Getting Writing Done, where I relate to writing as this big Thing one needs to do, and it starts feeling hard…

…even though, if I just stopped to explain what I wanted to express to another person, it would involve much less effort and be of a higher quality. (Call this Explaining To A Person.)

Why does this happen?

My guess is that Getting Writing Done often fails because one doesn’t have a target audience in mind. When I am Explaining To A Person, I have some expectation of the kinds of arguments this person will understand, and what will make sense to them. I can then query my intuitive model of “what do I need to explain for this person to understand”, while also tracking my sense of “does my current explanation feel like it is understandable to this person”.

In my previous post, I mentioned that one problem with Getting Work Done is that one becomes disconnected of any internal signals of quality. If I start off from the expectation that I am doing something Neat, then I can kind of hill-climb on that sense of Neatness - keep going in any direction that continues to feel equally or more Neat. Whereas if I just start forcing myself to do work because I should and I’m not optimizing for what feels valuable, there isn’t necessarily any point where my mind would start finding it valuable.

Likewise, if I start imagining a specific person who would read my writing, I can start with an explanation that would make sense to them, and have my mind generate writing that continues to feel like that. It often also works to imagine a more general audience - thinking that I’m writing something “to my Facebook friends” will produce a kind of gestalt feeling of everyone on Facebook who reacts to my posts, that also serves the same function.

In my experience, the choice of target audience also changes the writing. If I imagine that I will post something primarily to my Facebook audience, it will produce a different style than if I am writing something primarily to my blog, or primarily to LessWrong, or primarily to Twitter. This is shaped by my previous history of posting content on that platform - I will feel instinctively drawn toward the kind of content and argumentation that was well-received there before.

This might seem obvious - of course, your choice of target audience affects your writing choices! But while this involves some degree of conscious thinking about what I should make explicit and what I shouldn’t, it’s only to some degree. Much more seems to happen just from having the felt sense of a particular audience in my mind, and then the word-production systems in my brain automatically adjust the “shape” of the text that I want to produce.

Hugo Mercier & Dan Sperber’s influential 2011 paper “Why do humans reason” holds that the process of reasoning - which they define as the process of justifying and giving reasons for particular beliefs - evolved for the function of convincing others, and for spotting flaws in the arguments of others. This doesn’t mean that reasoning would necessarily be hostile. Suppose that I have in mind a course of action that I think would benefit us both. It is then in our shared interest that I try to convince you to join in by offering reasons for my belief that make sense to you. And it is also in our shared interest that you evaluate my argument for flaws and try to poke holes in it, so that we don’t pursue my thing if I happen to be wrong about it being a good idea. Similar to the prosecution and defense in a court arguing for their respective positions, taking up adversarial roles should ideally lead to a process that is truth-seeking overall.

One of the experimental findings that Sperber & Mercier discussed involved asking people to explore views like “What are the causes of school failure?” or “Would restoring the military draft significantly increase America’s ability to influence world events?”. The general finding was that people will produce pretty shallow and superficial reasoning in support of their thinking. However, if they then have to debate their view and be challenged on it, they will show significant improvement in their arguments.

Mercier & Sperber interpret this to mean that arguments for given positions are produced on an as-needed basis, and that this is rational behavior if the purpose of arguments is to convince others. If I offer a superficial argument and you believe it - maybe because you know me and trust me to know what I’m talking about, even if I don’t always explain it in detail - then it would be a waste of time for me to think about a more detailed argument. I only need to generate a better argument if you question it. However, I should still focus on the specific objections that you raised, rather than bother with hypothetical objections someone else might make.

The paper is full of these fascinating kinds of reframings of how seeming failures of rationality make sense if you see reasoning being primarily social rather than individual. One overall sense that emerges from it is that argumentation always needs to be tailored to a particular recipient - the arguments that will convince my grandmother may not convince my friend from school - and that the reasoning machinery in our minds is shaped to do this tailoring automatically.

Unless, of course, I don’t have any clear target audience in mind, and am just Getting Writing Done because I’m “supposed to write something”. In that case, it makes sense that the writing would feel laborious and unrewarding, since that reasoning machinery lacks some of the decision criteria it has evolved to use.

Discuss

Courtship Confusions Post-Slutcon

26 ноября, 2025 - 23:03

Published on November 26, 2025 8:03 PM GMT

Going into slutcon, one of my main known-unknowns was… I’d heard many times that the standard path to hooking up or dating starts with two people bantering for an hour or two at a party, lacing in increasingly-unsubtle hints of interest. And even in my own imagination, I was completely unable to make up a plausible-sounding conversation which would have that effect. Certainly it had never played out that way for me, nor had I personally witnessed it play out for anyone else.

So at slutcon itself, I volunteered for a session in which a few guys would be paired up with women and flirt on stage, with the session moderator giving live feedback and pointers. I was hoping I would finally be able to figure out how the heck a conversation ever leads to sex[1].

Within five minutes, the session moderator declared that I was already sufficiently witty that he had nothing to teach me, and he went fishing for a new volunteer who would be more instructively inept. I kid you not: I was booted from a flirting workshop for being too good at flirting, despite being completely unable to even imagine any plausible way for a banterous conversation to lead to hooking up. This certainly did not leave me less confused.

I left slutcon still unable to picture how banterous conversation leads to sex, increasingly skeptical that it does at all (or maybe that it works ridiculously inefficiently but most people have no better options), and increasingly confused that so many people seem so thoroughly convinced that it does.

What I did learn at slutcon was a lot more skill and confidence at physical escalation. I was already a pretty good dancer and leaned on that heavily, so this clearly isn’t the typical case, but multiple times now the path between “Want to dance?” and outright sex has involved less than five total minutes of talking, most of which is logistics. She’s very turned on by the dancing, there’s just not much reason for flirtatious talk to be involved at all. The one spot where flirtatious talk is useful is in spelling out exactly what I want to do to her, right before inviting her someplace more private. Whatever’s going on with the whole (banterous conversation) -> (sex/dating) process, I seem to be sidestepping it entirely, and whatever path I’m taking seems rather dramatically more efficient.[2]

… yet I still want to understand what the heck I’m sidestepping. What is going on with this whole banter -> sex thing?

Part Of A Model

One probably-relevant model-chunk which already makes sense was recently well articulated in The Status Game. Briefly, “the status game” of sex and seduction is roughly:

For males, coaxing females into sex grants status. The stingier the females are sexually, the more status the sex grants, because sex with a more celibate woman shows the guy overcame a greater challenge.
For females, getting males to put lots of effort into seduction grants status. The more effort the males put in, the more status the woman gains, because the men's effort is a costly signal of the woman’s appeal.

To be clear, this game sucks and I am quite happy to sidestep it. (One piece of anonymized feedback I got from slutcon was “I wanted to sleep with John but he didn’t pursue hard enough”, to which I say “lol sounds like you’ll just have to go fuck yourself”. I was surrounded by self-proclaimed sluts, why on earth would I waste such an opportunity on someone who’s going to make my life pointlessly difficult?)

The equilibrium here involves men pouring huge amounts of effort and resources into pursuit, women holding out for huge amounts of effort and resources expended in their pursuit, and people generally not having very much sex. And that indeed matches my impression of most short term mating markets: the ratio of (men’s effort in) to (sex out) seems crazy high for most hookups in most environments, it is just not worth it, unless a guy is really into this particular status game as a game in its own right (and doesn’t actually care that much about the sex per se).

But it does seem like this particular game is the standard.

So one hypothesis I’m tracking is that the whole banter -> sex process is standard for people who play the status game. And when people tell me that the banter -> sex process “works”, they mean that if you spend hundreds of hours flirting at house parties, you might get laid a handful of times. In other words, relative to my current standards, it basically just does not work.

Another piece of supporting evidence for this model: slutcon included a panel of guys with body counts over 100. While I didn’t attend that one, the main takeaway I heard from others is that every single one of those guys was very good at some touch-heavy activity like partner dance or massage, and that activity was pretty central to their sexual success.

… but even assuming that model is correct, it still doesn’t fully explain my observations. Why banterous conversation specifically, at parties specifically, as opposed to some other form of courtship? Why is that particular ineffective method the standard, rather than some other ineffective method?

An Alternative Submodel

An obvious theme in many womens' sexuality is that consent is anti-sexy[3]. Indeed, that's an understatement: even giving a hint that she actually wants the experience seems to be a major turn off for an awful lot of women. This creates an obvious conundrum for guys who want to distinguish that case from women who are in fact not interested.

I'm not going into detail on that model here, because its implications for flirtation are similar to the status game for current purposes. But I do want to note that a very different underlying model gives rise to somewhat similar conclusions, and this is another hypothesis I'm tracking.

Another Part Of A Model

You know what’s notoriously effective for hookups? Alcohol. Also, less notorious but pretty similar in effect, sleep deprivation. Getting drunk and staying up very late at a party are definitely not good for the quality of sex, but they are very much the standard, because they get people to drop their inhibitions. In particular, alcohol and sleep deprivation get people to pay less attention to the status game.

So another hypothesis I’m tracking is that the whole banter -> sex pipeline is not really about the banter at all. It’s just that people who are getting drunk and staying up late at a party tend to be banterous, and those having fun banterous conversations at parties are much more likely to grab another drink and stay up later. On this model, the game is not actually about the conversation per se; the main point of the conversation is just to kill time, wait for the booze and the sleepiness to do their job for everyone involved.

Notably, this model does predict a lot more of the specifics of the “standard” path: the banter, the parties. It also generalizes well to bars and clubs. It explains a lot of my apparent confusion: I don’t drink and am relatively careful to get plenty of sleep, so I am almost never in those sorts of environments. This model also matches common wisdom about the rarity of sober hookups.

… yet people still tell me that the banter pipeline applies even in the absence of booze. The booze/sleepiness model nails down enough predictions that I’m pretty skeptical of the people telling me something else is central, but I do still find it plausible that I’m missing substantial noncentral pieces.

So what am I still missing?

^
… or dating, but this being slutcon the focus was obviously on sex, and this post will mostly focus on sex.
^
Also, about half of my sexual/romantic partners have been women who very unambiguously approached me, e.g. explicitly asked me out, without me making any particular effort to court them beforehand. That’s another way to sidestep whatever the standard process is, and I definitely endorse it strongly.
^
... though anxiety is also anti-sexy, and more explicit consent usually reduces anxiety a lot. This leads to a lot of confusion amongst people who are not separately tracking "absence of anti-sexy things" and "presence of sexy things".

Discuss

Books cover a larger idea-space than movies

26 ноября, 2025 - 22:55

Published on November 26, 2025 7:55 PM GMT

Note: I'm writing every day in November, see my blog for disclaimers.

I’m hoping that multi-modal embedding models (which convert videos, images, and text into points in a high-dimensional space such that a video of a dog, a photo of a dog, and text describing a dog all end up close together) will allow this statement to be formalised cheaply in the near future.

If you consider the “range” of ideas that are conveyed in books, and the range of ideas that are conveyed in movies, I certainly have the intuition that books seem to be much broader/weirder/different in the ideas they describe when compared to movies. Movies, by comparison, seem muted and distinctly more “normal”.

I believe this to be due to how these two media come to be: movies are enormously expensive, and require the creative input of dozens to ~100s of people (writers, actors, producers, sound & music, lighting, video, VFX, set design, the list goes on). If a screenwriter has a complicated idea and wants it to appear in a film, each of these people has to understand (and importantly, care) about this idea for it to actually arrive on-screen in its original form. This puts an upper bound on how “weird” an idea can be for it to be on the big screen.

I’m less informed about the production of books, but I’m struggling to imagine more than 30 people who could have an impact on the creative aspects of a book. As an upper bound, maybe you have multiple authors, multiple editors, and several people from the publisher. But as a lower bound, I imagine you could write a book with creative input from just the author and the editor. If an author has a unusual idea they want to convey, they just have to make sure their one editor doesn’t dilute the core paragraphs too much, and this idea can then go straight into the reader’s noggin.

You might quibble with my “100s of people” characterisation for movies, but that’s okay because this exact number doesn’t matter, it just matters that most movies involve more people than most books. Because the distinction isn’t really movies-vs-books, it’s about the number of people involved: The more people involved, the trickier it is to get slippery ideas through them all, and so the more “normal” your ideas have to become.

You can see this by looking at low-headcount “movies” (YouTube videos, indie films, short films) which tend to be more out-there and weird, and also by looking at high-headcount “books” (pop-sci, magazines, newspapers) which tend to have more in-distribution ideas. You can even expand this to other creative media such as podcasts, theatre, poetry.

Creative media are ideal for doing this sort of headcount-vs-overton-window analysis, because there’s typically very little that fundamentally constrains what the message can be in creative media.

The headcount-overton-window tradeoff also applies to startups, organisations, political campaigns, public movements, although these have more constraints than creative media, such as profitability, existing in the physical world, convincing investors. But there’s a reason why “donate to save this puppy” is an easier pitch than “donate to save these shrimp”.

Discussing memetics is always tricky, but I do believe this tradeoff is real, and that it effects the objects we see in the world. We can draw informative conclusions from this tradeoff: to find unique ideas, you must consume content that required very few people’s approval in order to exist (e.g. blogs, preprints, off-the-cuff discussions with smart people). To produce unique ideas, you will be fighting an uphill battle if you subject that idea to other’s opinions. Sometimes this battle is worthwhile; I said the ideas would be unique, not good. But every novel idea you introduce is something that needs to fight for space in someone’s mind, and you’ll need to convince them not to project your idea onto their existing map of the world. Every introduction is a chance for failure of communication (which may go unnoticed by you). But also, every introduction is a chance for someone to cull your bad ideas.

So when you feel in a rut, and lacking interesting thoughts, consider the media you consume. Are you sampling from the same distribution as everyone else? By how much has the media you consume been filtered (either anthropically or algorithmically)? Books, for me, are second to none for novel ideas, and I look forward to being able to demonstrate this empirically.

Discuss

The Big Nonprofits Post 2025

26 ноября, 2025 - 22:00

Published on November 26, 2025 7:00 PM GMT

There remain lots of great charitable giving opportunities out there.

I have now had three opportunities to be a recommender for the Survival and Flourishing Fund (SFF). I wrote in detail about my first experience back in 2021, where I struggled to find worthy applications.

The second time around in 2024, there was an abundance of worthy causes. In 2025 there were even more high quality applications, many of which were growing beyond our ability to support them.

Thus this is the second edition of The Big Nonprofits Post, primarily aimed at sharing my findings on various organizations I believe are doing good work, to help you find places to consider donating in the cause areas and intervention methods that you think are most effective, and to offer my general perspective on how I think about choosing where to give.

This post combines my findings from the 2024 and 2025 rounds of SFF, and also includes some organizations that did not apply to either round, so inclusion does not mean that they necessarily applied at all.

This post is already very long, so the bar is higher for inclusion this year than it was last year, especially for new additions.

If you think they are better places to give and better causes to back, act accordingly, especially if they’re illegible or obscure. You don’t need my approval.

The Big Nonprofits List 2025 is also available as a website, where you can sort by mission, funding needed or confidence, or do a search and have handy buttons.

Table of Contents

Organizations where I have the highest confidence in straightforward modest donations now, if your goals and model of the world align with theirs, are in bold, for those who don’t want to do a deep dive.

A Word of Warning

The SFF recommender process is highly time constrained, and in general I am highly time constrained.

Even though I used well beyond the number of required hours in both 2024 and 2025, there was no way to do a serious investigation of all the potentially exciting applications. Substantial reliance on heuristics was inevitable.

Also your priorities, opinions, and world model could be very different from mine.

If you are considering donating a substantial (to you) amount of money, please do the level of personal research and consideration commensurate with the amount of money you want to give away.

If you are considering donating a small (to you) amount of money, or if the requirement to do personal research might mean you don’t donate to anyone at all, I caution the opposite: Only do the amount of optimization and verification and such that is worth its opportunity cost. Do not let the perfect be the enemy of the good.

For more details of how the SFF recommender process works, see my post on the process.

Note that donations to some of the organizations below may not be tax deductible.

A Note To Charities

I apologize in advance for any errors, any out of date information, and for anyone who I included who I did not realize would not want to be included. I did my best to verify information, and to remove any organizations that do not wish to be included.

If you wish me to issue a correction of any kind, or to update your information, I will be happy to do that at least through the end of the year.

If you wish me to remove your organization entirely, for any reason, I will do that, too.

What I unfortunately cannot do, in most cases, is take the time to analyze or debate beyond that. I also can’t consider additional organizations for inclusion. My apologies.

The same is true for the website version.

I am giving my full opinion on all organizations listed, but where I feel an organization would be a poor choice for marginal dollars even within its own cause and intervention area, or I anticipate my full opinion would not net help them, they are silently not listed.

Use Your Personal Theory of Impact

Listen to arguments and evidence. But do not let me, or anyone else, tell you any of:

What is important.
What is a good cause.
What types of actions are best to make the change you want to see in the world.
What particular strategies are most promising.
That you have to choose according to some formula or you’re an awful person.