Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 26 секунд назад

Book review: The Geography of Thought

9 февраля, 2021 - 21:47
Published on February 9, 2021 6:47 PM GMT

Book review: The Geography of Thought: How Asians and Westerners Think Differently... and Why, by Richard E. Nisbett.

It is often said that travel is a good way to improve one's understanding of other cultures.

The Geography of Thought discredits that saying, by being full of examples of cultural differences that 99.9% of travelers will overlook.

Here are a few of the insights I got from the book, but I'm pretty sure I wouldn't have gotten from visiting Asia frequently:

There's no Chinese word for individualism - selfish seems to be the closest equivalent.

Infants in the US are often forced to sleep in a separate bed, often in a separate room. That's rather uncommon in Asia. Does this contribute to US individualism? Or is it just a symptom?

There are no Asians in Lake Wobegon. I.e. Asians are rather reluctant to rate themselves as above average.

Westerners want contracts to be unconditionally binding, whereas Asians want contracts to change in response to unexpected contexts.

Asians are likely to consider justice in the abstract, by-the-book Western sense to be rigid and unfeeling.

Chinese justice is an art, not a science.

Origins of Western Culture

Those cultural differences provide hints about why science as we know it developed in the West, and not in Asia.

I read Geography of Thought in order to expand my understanding of some ideas in Henrich's WEIRDest People.

Nisbett disagrees somewhat with Henrich about when WEIRD culture arose, writing a fair amount about the Western features of ancient Greek culture.

Nisbett traces some of the east-west differences to the likelihood that the Greeks met more apparent contradiction than did Asians, via trade with other cultures. That led them to devote more attention to logical thought. (Here's an odd claim from Nisbett: ancient Greeks were unwilling to adopt the concept of zero, because "it represented a contradiction").

Nisbett agrees with Henrich that there was some sort of gap between ancient Greek culture and the Reformation, but believes the gap came later than Henrich does. These two quotes are about all that Nisbett has to say about the gap:

As the West became primarily agricultural in the Middle Ages, it became less individualistic.

The Romans brought a gift for rational organization and something resembling the Chinese genius for technological achievement, and - after a trough lasting almost a millennium - their successors, the Italians, rediscovered these values ... The Reformation also brought a weakened commitment to the family and other in-groups coupled with a greater willingness to trust out-groups

Neither Nisbett nor Henrich convinced me that they know much about any such period of reduced individualism - they don't seem to consider it important.

Reductionism and Categorization

I used to interpret attacks on reductionism as attacks on a valuable aspect of science. I now see an alternate understanding: a clash of two cognitive styles, reflecting differing priors about how much we can usefully simplify our models of the world.

The Western goal of finding really simple models likely helped generate the study of physics. I'm guessing it also contributed a bit to the West's role in eradicating infectious diseases.

However, it may have been counter-productive at dealing with age-related diseases. Let's look at the example of Alzheimer's.

Western researchers have been obsessed with the simple model of beta amyloid being the sole cause of the disease. Drugs targeting beta amyloid have been failing at a rate that is worse than what we should expect due to random chance if they were placebos. Yet some researchers still pursue drugs that target beta amyloid.

Some of that focus on single causes is due to the way that medical research depends on patents, but don't forget that patent law is a product of Western culture.

Meanwhile, outside of the mainstream, there are some signs of progress at treating Alzheimer's using approaches that follow a more holistic cognitive style. They posit multiple, overlapping factors that contribute to dementia, and entertain doubts about how to classify various versions of dementia.

I also see some hints that traditional Asian medicine has done better than mainstream Western medicine at treating Alzheimer's. The results still seem poor, but the risk/reward ratio seems good enough that I'm trying a few of them.

High modernism, combined with excessive reification of categories, may have led the medical establishment on some dead-end paths.

In addition, Western medicine has been much more eager to adopt surgery than China - presumably due to an expectation that cutting out "the cause" of a disease will cure it. I'm moderately confident that Western medicine does too much surgery. I don't have any guess about whether Asian cultures do too little.

Chuang Tzu is quoted as saying, "Classifying or limiting knowledge fractures the greater knowledge."

it's been suggested that the distinction between "human" and "animal" insisted upon by Westerners made it particularly hard to accept the concept of evolution. ... Evolution was never controversial in the East because there was never an assumption that humans sat atop a chain of being and had somehow lost their animality.

Westerners needed to overcome the habit of classifying humans and animals as categories with different essences. Asians are much less comfortable with attaching importance to categories and essences, so evolution required less change in their worldviews.

Doesn't the Western lead in reductionist science conflict with the evidence of Asian students doing well at math and science? Nisbett says that's partly explained by Asians working harder:

due at least in part to the greater Western tendency to believe that behavior is the result of fixed traits. Americans are inclined to believe that skills are qualities you do or don't have, so there's not much point in trying to make a silk purse out of a sow's ear. Asians tend to believe that everyone, under the right circumstances and with enough hard work, can learn to do math.

There's some important tension between this and the message of The Cult of Smart. Nisbett tells us that American math-teaching isn't as good as the Asian version. But that can't be the full answer - Cult of Smart indicates that schools rejected key elements of Western culture in the past few decades. Key parts of that trend happened just before Geography of Thought was published.

So the US seems to be adopting parts of Asian culture that make schools more cruel, and more effective at producing excellent graduates. But that trend seems unstable, due to the delusion that it's promoting the Western ideal of equality.


For a long time, I believed that the Raven's Progressive Matrices Test was culture-neutral.

Nisbett compares an example from a CFIT test (like Raven's, but with "culture fair" in the name) with an example from an IQ-like Chinese test. The Chinese test is more focused on relationships between parts. It was easy for me to see that the two tests were optimized for mildly different notions of intelligence, so I was unsurprised when Nisbett reported that Chinese subjects showed higher scores on the Chinese test, and Americans showed higher scores on CFIT.

I'm a bit frustrated that Nisbett is vague about the magnitude of the differences, and that he cites only an unpublished manuscript that he co-authored. Publish it now, Nisbett!

Both notions of intelligence seem quite compatible with common notions of smartness, differing only in which skill subsets ought to be emphasized most. So this isn't like the usual commentary on bias in IQ tests that's looking for an excuse to reject intelligence testing.


I previously wrote:

I'm surprised to find large differences in how much various cultures care about distinguishing intentional and accidental harm, with WEIRD people caring the most, and a few cultures barely distinguishing them at all.

Nisbett hints that some of that is due to the WEIRD expectation that actions have a single cause, and can't result from a combination of intentional and accidental factors. Some of it might also be due to Westerners doing more causal attribution in general.

Virtue Ethics

I wonder how cultural differences affect attitudes toward ethics?

In particular, I wonder whether Asian cultures care less about virtue ethics, due to less influence from Fundamental Attribution Error?

Some hasty research suggests that the answers are controversial.

The Stanford Encyclopedia of Philosophy says:

What makes the characterization of Confucianism as a virtue ethic controversial are more specific, narrower senses of "virtue" employed in Western philosophical theories. Tiwald (2018) distinguishes between something like the broad sense of virtue and a philosophical usage that confers on qualities or traits of character explanatory priority over right action and promoting good consequences. Virtue ethics in this sense is a competitor to rule deontological and consequentialist theories. There simply is not enough discussion in the Confucian texts, especially in the classical period, that is addressed to the kind of questions these Western theories seek to answer.

There are other narrower senses of "virtue" that are clearly mischaracterizations when applied to Confucian ethics. Virtues might be supposed to be qualities that people have or can have in isolation from others with whom they interact or from their communities, societies, or culture. Such atomistic virtues could make up ideals of the person that in turn can be specified or realized in social isolation. ... influential critics of the "virtue" characterization of Confucian ethics ... seem to be supposing that the term is loaded with such controversial presuppositions.


Geography of Thought is a great choice if you want to understand the cultural differences between the US and China. It complements WEIRDest People fairly well.

Geography of Thought is mostly about two sets of cultures, with little attention to cultures other than those of eastern Asia and the West.

Nisbett seems a bit more rigorous than Henrich, but Henrich's cultural knowledge seems much broader. Geography of Thought doesn't quite satisfy the "and Why" part of it's subtitle, whereas Henrich makes an impressive attempt at answering that question.


No title

9 февраля, 2021 - 19:00
Published on February 9, 2021 4:00 PM GMT

Still Not in Charge

Epistemic Status: Speed premium, will hopefully flesh out more carefully in future

Previously: Why I Am Not in Charge

A brief update about my exchange with Scott from this past week.

After Scott Alexander wrote a post about why WebMD is terrible and it would be good but impossible to put me in charge of the CDC, which is super flattering and unsurprisingly blew up my inbox, I wrote a quick response post to expand on things, and give more color on my model of the dynamics involved in organizations like the CDC and FDA, my view of how people in my position can often get reliably ahead of such organizations, and what would happen if one tried to change them and get them to do the most useful things. That required tying things back to several key past posts, including Zeroing Out, Leaders of Men, Asymmetric Justice, Motive Ambiguity, and the sequences on Simulacra and Moral Mazes.

The disagreements between my model and Scott’s, and the places in which my communication attempts fell short, are broadly in two (closely linked) categories, which his Reddit response captured well and made clearer to me, as did other reactions on the same thread.

The first category is where claims about how perverse things are get rounded down and not seen. Scott is advocating here for what we might call the Utility Function Hypothesis (UFH). 

The second is (as Scott explicitly notes) the generalized form of the Efficient Market Hypothesis, which one might call the Efficient Action Hypothesis (EAH). 

UFH and EAH are linked.

If UFH is impactfully false, then EAH is also false. If you’re acting meaningfully distinctly from how you would act with a coherent utility function, you are leaving money on the table.

If UFH is effectively true (e.g. it is not impactfully false), then EAH is plausible, but can still either be true or false. EAH could still be false if either one could have a better model of the impact of decisions than those making the decisions (this could be better political sense, better physical-impact sense that then impacts the political calculus, or both). EAH could also still be false if the utility function being maximized doesn’t square up with what we’d mean by politically successful.

If the EAH is true, then UFH is effectively true as well. Anyone acting in a way that can’t be improved upon isn’t meaningfully different from how they’d be with a utility function.

If the EAH is false, then that’s strong evidence that UFH is also false (since EAH -> UFH), but UFH could still be true if the EAH is false due to our having superior information, especially superior information about physical impacts.

On the first disagreement, I think we can look at the decisions in detail and find evidence for who is right. The utility function hypothesis expects to find sensible trade-offs being made, with bad decisions only being made when good information was unavailable or political pressure was stronger than the physical stakes. We can ask ourselves if we see this pattern. The difficulty is that the political pressures are often invisible to us, or hard to measure in magnitude. But if it’s a matter of battling real interests, we should expect political pressure to generally push in favor of useful actions for particular interests, rather than for perversity in general. 

Another way of looking at the question is, do such folks seem to be goal optimizers or adaptation executors? To the extent that actions seem to reflect goals, including political ones, what kind of time horizon and discount rate seem to be in play? 

We can also ask whether making destructive decisions seems to correlate with political profit. The issue here is that both sides have competing hypotheses that explain this correlation. The UFH says this is because of trade-offs. The alternate hypothesis says that this happens because there has been selection for those who act as if they need to be careful not to be seen favoring the constructive over the destructive because it is constructive.  

What distinguishes these claims is that the UFH thinks that when the explicit political stakes are low then constructive decisions will dominate, whereas the alternate hypothesis thinks that there can be no explicit stakes at all and the destructive decisions still happen, because the mechanism causing them is still functioning. 

On the second disagreement, my claim is that if you could execute with an ordinary level of tactical political savvy and not make continuous unforced errors, except when it mattered you used my judgment to make constructive decisions and implement constructive policies, that this would have a good chance of working out, especially if applied to the FDA along with the CDC.

My claim is importantly not that putting me in as CDC director tomorrow would have a good chance of working out, which it most certainly wouldn’t because I wouldn’t have the ordinary skill in the art necessary to not have the whole thing blow up for unrelated reasons.

But again, I don’t need to be a better overall judge of what is politically savvy and be able to execute the entire job myself to point to a better path, I only need to make improvements on the margin, even if that margin is relatively wide and deep. I can claim the EAH is importantly false and point to big improvements without being able to reimplement the rest of the system.

In particular, don’t make me head of the FDA, that’s crazy, but do appoint my father Solomon Mowshowitz as head of the FDA, give us some rope, and watch what happens. Scott was talking about the director of the CDC, which I’d also accept, but I think you can have a lot more impact right now at the FDA. 

Why do I think a lot of politicians are leaving money on the ground for ‘no reason’, other than “Donald Trump spent four years as President of the United States’? 

First, my model is that politicians mostly aren’t goal maximizers with utility functions, they’re adaptation executors who have developed systems that have been shaped to seek power.  Those adaptations lead to political success. They’ve been heavily selected for those attributes by people looking for those attributes. One of the adaptations is to be visibly part of this selection process. Another is avoiding displaying competing loyalties, such as caring about money being on the ground enough to both see the money on the ground and then pick it up. 

Second, the politicians don’t directly know what would and wouldn’t work out, and have been selected for not thinking it is possible to know such things. To the extent they try to do things that would work out, they approximate this by having a mechanism that avoids being blamed for things within the next two weeks, which is then subject to back propagation by others. If you do something with bad consequences next year or next month, if it’s bad enough the hope is that other people notice this and get mad about it now, you notice that and so you choose differently. The advantage of this approach is that Avoid Blame is a Fully General Excuse for action (or even better, for inaction), so it doesn’t cause suspicion that you prefer constructive to destructive action, or think you can tell the difference. 

Third, this is all outside of the training data that the politicians learned on. Everyone involved is trained on data where the feedback loops are much longer, and the physical impacts are slow and far away. It hasn’t sunk in that this is a real emergency, and that in an emergency the rules are different. One can think of this as a battle between perception and reality, to see who can get inside whose OODA loop. People in mazes (and everyone involved here is in a maze) are used to making destructive decisions and then either not having the consequences rebound back at all, or being long gone before the physical consequences can catch up with them. Also, a lot of these learned behaviors go back to previous times without rapid social media feedback loops and other ways for us to distinguish constructive from destructive action, or identify lies or obvious nonsense. Back then there was more elite control over opinion and more reason to worry greatly about being a Very Serious Person who properly demanded their Sacrifices to the Gods, and consequences were less likely to back propagate into current blame and credit assignments. 

Fourth, Leaders of Men, but also imposter syndrome and the illusion of competence. Everyone is everywhere and always muddling through far more than anyone realizes. Always have been. They make dumb mistakes and unforced errors, they overlook giant opportunities, they improvise and act confident and knowledgable. How many political campaigns does one need to watch unforced error after unforced error, to say ‘how did we end up with all these horrible choices and no good ones?’ and to watch candidates be woefully unprepared time and time again, before one realizes that this is the normal state of affairs? We’re choosing the people who were in the right place at the right time with the skills that most impact ability to raise money and campaign, not the people who are the best at governance. There is a dramatic difference in effectiveness levels between different politicians and leaders, not only in government but also in other areas. You take what you can get. And again, we’re aiming at improving on the margin, and it would be pretty shocking if there weren’t large marginal improvements available that we could spot if we tried. 

Fifth, because they’re not properly modeling the shifts that occur when policy changes. The people who get to move elite opinion, and hence move ‘expert’ opinion, don’t realize this is within their power. Again, each time there was a fight over a shift from a destructive policy stance or claim to a constructive one in the pandemic, once the shift was made, most of the ‘experts’ saying otherwise fell in line immediately. At maximum, they nominally complained and then kept quiet. It’s almost like they’re not offering their expertise except insofar as they use it to back up what elites decided to tell us.  

It’s an interesting question to what extent that mechanism doesn’t work when the new decision is destructive, but again we have data on that, so think back and form your own opinion on who would push back how much on such questions. 

You could also respond that the constructive changes were chosen and timed exactly in order to be politically beneficial, and thus this isn’t a fair test. There’s certainly some selection effects but if you compare the results to a naive prior or to the prediction of the trade-off model, I think you’ll notice a big difference.

Sixth, because bandwidth is limited. Politicians aren’t looking on the sidewalk for the bill, so they don’t notice it and therefore don’t pick it up. When you are a powerful person there’s tons of things to do and no time to do them, whatever your combination of avoiding blame, executing adaptations, cashing in, gathering power and trying to do the most good as you see it. Everyone trying to contact you and give you ideas has an agenda of some form. Getting the good ideas even on the radar screen is hard even when you have a relatively competent and well-meaning group of people. 

Seventh, this is a known blind spot, where there is reliably not enough attention to satisfying the real needs of voters and giving them what they care about, and not doing this reliably loses people power, while satisfying such needs reliably gets rewarded. This is true for things that voters are right about, and also things voters are wrong or selfish about. 

Eighth, if a process filters out actions by making them unthinkable to anyone with the power to execute on them, partly by filtering who gets power and partly by getting those who seek power to self-modify, and thus such people never think seriously about them, or have lots of people whose job it is when they accidentally do think them to point out how unthinkable they are via what are usually essentially Beware Trivial Inconveniences arguments, it’s hard to turn around and call not taking those actions evidence that those actions wouldn’t work if someone actually did them.

Lastly, because ‘there’s a good chance this would actually work out’ does not translate to free money on the ground. The political calculus is not ‘free money’ here, it’s ‘if this works sufficiently well, you reap large political benefits that outweigh your costs.’ You’d be betting on an alternate mechanism kicking in. Doing a different thing that isn’t as legible puts one very open to large amounts of blame via Asymmetric Justice, and inherently feels terrible for those trained on such data. None of this looks like a safe play, even if a lot of it on the margin is safe. Doing the full throttle version would definitely not be safe. 

In general, rather than look at this as a ‘all trading opportunities are bad or someone would have taken them already’ or ‘if the fence should have been taken down then the fence-removal experts would have already taken care of it,’ look at this as the “How are you f***ing me?” or Chesterson’s Fence question. Before you take down a fence you need to know why someone put it up. Before you do a trade, you need to know the reason why you have the opportunity to do this trade. We see politicians failing to do seemingly obviously correct things that would be physically beneficial to people and look like they would be popular and net them political capital, so we need an explanation for why they’re not acting on it.

We have plenty of interlocking explanations for why they’re not acting on it. That doesn’t mean that any given instance doesn’t have additional good explanations, including explanations that could plausibly carry the day. And some of the explanations given here are reasonable reasons to not do some of the things, including the pure ‘there are people who don’t want you to do that, and they pressure you, and that sucks and raises the cost of acting an amount it’s hard for us to measure.’  

As for the pure modesty argument, that I am not a political expert and thus shouldn’t expect to be able to predict what will win political capital, the response is in two parts.

First, I’m also not a medical or biological expert, yet here we are. I fully reject the idea that smart people can’t improve on the margin on ‘expert’ opinion, period. Welcome to 2021. Modesty shmodesty. 

Second, much of the difference is in our physical world models and not our political models. To fully model politics, one must fully understand the world. 

I don’t think this fully did justice to the questions involved. That will require several posts that I’ve been working on for a while in one form or another and are difficult to write. This did make writing those posts easier, so there is hope.


We Need Browsers as Platforms

9 февраля, 2021 - 18:40
Published on February 9, 2021 3:40 PM GMT

It's fashionable to say that the web is bloated, and that the features built to support webapps make it too complex. You can divide the web into:

  • Documents: providing information. News, blog posts, documentation.

  • Apps: doing things. Email, spreadsheets, games.

(This is really a continuum, where a blog post with a comment section is pretty documenty but also a bit appy.)

On one hand, I completely agree that supporting apps makes the platform complex: browsers are incredibly complicated to build and work on, with such a high ongoing maintenance cost that we only have three rendering engines: Firefox's Gecko, Chrome's Blink, and Safari's Webkit (Blink is a fork of Webkit). On the other, supporting apps is much better than the alternatives.

Outside of a browser, there are essentially two models for getting applications:

  • Independent installation. You download the program for the manufacturers website, or loaded off a CD. The desktop model.

  • App store. Your OS has a list of programs that can be installed, and which have gone through some amount of review. The smartphone model.

Independent installation is decentralized, but also a security nightmare. Random users installing random software gives you botnets. App stores are centralized, which puts their operators in a position of enormous power over what users can run on their devices, and means governments can require them to take down apps.

The has developed with the principle that it should always be safe to visit a site. As new capabilities have been added this has been critical to maintain. This means you don't need an app store, with power to reject your app.

Ten years ago Mozilla posted Booting to the Web:

Mozilla believes that the web can displace proprietary, single-vendor stacks for application development. To make open web technologies a better basis for future applications on mobile and desktop alike, we need to keep pushing the envelope of the web to include—and in places exceed—the capabilities of the competing stacks in question.
We want to take a bigger step now, and find the gaps that keep web developers from being able to build apps that are—in every way—the equals of native apps built for the iPhone, Android, and WP7. The web platform has come so far in supporting apps over this decade; we couldn't have made Bucket Brigade without Web Audio or WebRTC. A web developer should be able to do anything native app developers can, making apps for any device, free from vendor veto.


Epistemology of HCH

9 февраля, 2021 - 14:46
Published on February 9, 2021 11:46 AM GMT


HCH is a recursive acronym meaning “Humans consulting HCH”. Coincidentally, It’s also a concept coined by Paul Christiano, central in much of the reasoning around Prosaic AI Alignment. Yet for many, me included, the various ways in which it is used are sometimes confusing.

I believe that the tools of Epistemology and Philosophy of Science can help understand it better, and push further the research around it. So this post doesn’t give yet another explanation of HCH; instead, it asks about the different perspectives we can take on it. These perspectives capture the form of knowledge that HCH is, what it tells us about AI Alignment, and how to expand, judge and interpret this knowledge. I then apply these perspectives to examples of research on HCH, to show the usefulness of the different frames.

Thanks to Joe Collman, Jérémy Perret, Richard Ngo, Evan Hubinger and Paul Christiano for feedback on this post.

Is it a scientific explanation? Is it a model of computation? No, it’s HCH!

HCH was originally defined in Humans Consulting HCH:

Consider a human Hugh who has access to a question-answering machine. Suppose the machine answers question Q by perfectly imitating how Hugh would answer question Q, if Hugh had access to the question-answering machine.

That is, Hugh is able to consult a copy of Hugh, who is able to consult a copy of Hugh, who is able to consult a copy of Hugh…

Let’s call this process HCH, for “Humans Consulting HCH.”

Nowadays, this is actually called weak HCH, after the Strong HCH post which extended the definition. That being said, I’m only interested in perspective about HCH, which includes the questions asked about it and how to answer them. Although the difference between Weak and Strong HCH matters for the answers, the questions and perspective stay the same. I’ll thus use HCH to mean one or the other interchangeably.

The main use of HCH is as an ideal for what a question-answerer aligned with a given human should be like. This in turn serves as the aim of schemes like IDA and Debate. But thinking about HCH entangles many different angles. For example, what can HCH do? One can interpret this question as “What is the power of the HCH scheme?” or “What questions can HCH answer for a given human?” or “Is HCH aligned with the human parametrizing it?” or “Is HCH competitive?”. Each one of these questions requires different assumptions and focus, making it hard to  grasp the big picture.

I claim that most of what is studied about HCH can be brought to order if we explicitly use different epistemological perspectives through which to see it. This is close to the paradigms of Kuhn in The Structure of Scientific Revolutions: a framing of the phenomenon studied which explains its most important aspects, how to study them, and what counts as knowledge for this approach. The perspectives I present are both more abstract than paradigms in natural sciences (they’re more epistemological paradigms) and less expansive. Yet I believe the intuition stays the same.

Kuhn writes that paradigms are necessary for what he calls normal science (and which encompass the majority of scientific research), which is solving the puzzles generated by the paradigm. Similarly, the perspectives I propose each put some questions and puzzles in front, and limit the scope of HCH somewhat. Thus not one is supposed to be sufficient; they all have something to bring to the table.

Each of these perspective provide assumptions about HCH:

  • What it is
  • What are the important questions about it

But before giving these, let’s start with a classical perspective in science that doesn’t work well here.

False start: HCH as explanation of a natural phenomenon

In natural sciences, ideas are often explanations of natural phenomena, like lightning and oxidation. Once armed with such an explanation, the research attempts among other things to check it against previous data of the phenomenon, and to predict new behavior for future experiments.

Of what could HCH be the explanation? In the original post, Paul describes it as

our best way to precisely specify “a human’s enlightened judgment” [about a question Q]

So the phenomenon is enlightened judgement. Yet this looks more like an ideal than a phenomenon already present in the world.

Indeed, Paul’s Implementing our considered judgement, his first post on the topic as far as I know, presents the similar notion of “considered judgment” as the outcome of a process that doesn’t exist yet.

To define my considered judgment about a question Q, suppose I am told Q and spend a few days trying to answer it. But in addition to all of the normal tools—reasoning, programming, experimentation, conversation—I also have access to a special oracle. I can give this oracle any question Q’, and the oracle will immediately reply with my considered judgment about Q’. And what is my considered judgment about Q’? Well, it’s whatever I would have output if we had performed exactly the same process, starting with Q’ instead of Q.

Seeing HCH as an explanation of enlightened judgment thus fails to be a fruitful epistemological stance, because we don’t have access to considered judgements in the wild to check the explanation.

HCH as philosophical abstraction

If enlightened judgment isn’t a phenomenon already existing in the world, intuitions nonetheless exist about what it means. For example, it feels like an enlightened judgment should depend on many different perspectives on the problem instead of only on the most obvious one. Or that such judgment shouldn’t change without additional information.

This leads to the perspective of HCH as a philosophical abstraction of the fuzzy intuitions around enlightened judgment (on a question Q). The aim of such an abstraction is to capture the intuitions in a clean and useful way. We’ll see a bit later for what it should be useful for.

How should we judge HCH as a philosophical abstraction of enlightened judgement? One possible approach is inspired by Inference to the Best Explanation with regard to intuitions, as presented by Vanessa in her research agenda:

Although I do not claim a fully general solution to metaphilosophy, I think that, pragmatically, a quasiscientific approach is possible. In science, we prefer theories that are (i) simple (Occam's razor) and (ii) fit the empirical data. We also test theories by gathering further empirical data. In philosophy, we can likewise prefer theories that are (i) simple and (ii) fit intuition in situations where intuition feels reliable (i.e. situations that are simple, familiar or received considerable analysis and reflection). We can also test theories by applying them to new situations and trying to see whether the answer becomes intuitive after sufficient reflection.

In this perspective, the intuitions mentioned above play the role of experimental data in natural sciences. We then want an abstraction that fits this data in the most common and obvious cases, while staying as simple as possible.

What if the abstraction only fit some intuitions but not others? Here we can take note from explanations in natural sciences. These don’t generally explain everything about a phenomenon -- but they have to explain what is deemed the most important and/or fundamental about it. And here the notion of “importance” comes from the application of the abstraction. We want to use enlightened judgement to solve the obvious question: “How to align an AI with what we truly want?” (Competitiveness matters too, but it makes much more sense from the next perspective below)

Enlightened judgement about a question serves as a proxy for “what we truly want” in the context of a question-answerer. It’s important to note that this perspective doesn’t look for the one true philosophical abstraction of enlightened judgment; instead it aims at engineering the most useful abstraction for the problem at hand -- aligning a question-answerer.

In summary, this perspective implies the following assumptions about HCH.

  • (Identity) HCH is a philosophical abstraction of the concept of enlightened judgment for the goal of aligning a question-answerer
  • (Important Questions) These includes pinpointing of the intuitions behind enlightened judgment, weighting their relevance to aligning a question-answerer, and check that HCH follows them (either through positive arguments or by looking for counterexamples)
HCH as an intermediary alignment scheme

Finding the right words for this one is a bit awkward. Yes, HCH isn’t an alignment scheme proper, in that it doesn’t really tell us how to align an AI. On the other hand, it goes beyond what is expected of a philosophical abstraction, by giving a lot of details about how to produce something satisfying the abstraction.

Comparing HCH with another proposed philosophical abstraction of enlightened judgement makes this clear. Let’s look at Coherent Extrapolated Volition (CEV), which specify “what someone truly wants” as what they would want if they had all the facts available, had the time to consider all options, and knew enough about themselves and their own processes to catch biases and internal issues. By itself, CEV provides a clarified target to anyone trying to capture someone’s enlightened judgement. But it doesn’t say anything about how an AI doing that should be built. Whereas HCH provides a structured answer, and tells you that the solution is to get as closed as possible to that ideal answer.

So despite not being concrete enough to pass as an alignment scheme, HCH does lie at an intermediary level between pure philosophical abstractions (like CEV) and concrete alignment schemes (like IDA and Debate).

So what are the problems this perspective focuses on? As expected from being closer to running code, they are geared towards practical solutions:

  • How much can HCH be approximated?
  • How competitive is HCH (and its approximations)?

That is, this perspective cares about the realization of HCH, assuming it is what we want. It’s not really the place to wonder how aligned HCH is; knowing it is, we want to see how to get it right in a concrete program, and whether it costs too much to build.

In summary, this perspective implies the following assumptions about HCH.

  • (Identity) HCH is an intermediary alignment scheme for a question-answerer.
  • (Important Questions) Anything related to the realization of HCH as a program: approximability, competitiveness, limits in terms of expressiveness and power.
HCH as a model of computation

Obvious analogies exist between HCH and various models of computations like Turing Machines: both give a framework for solving a category of problems -- questions on one side and computable problems on the other. HCH looks like it gives us a system on which to write programs for question answering, by specifying what the human should do.

Yet one difficulty stares us in the face: the H in HCH. A model of computation, like Turing Machines, is a formal construct on which one can prove questions of computability and complexity, among others. But HCH depends on a human at almost every step in the recursion, making it impossible to specify formally (even after dealing with the subtleties of infinite recursion).

Even if one uses the human as a black box, as Paul does, the behavior of HCH depends on the guarantees of this black box, which are stated as “cooperative”, “intelligent”, “reasonable”. Arguably, formalizing these is just as hard as formalizing good judgment or human values, and so proves incredibly hard.

Still, seeing HCH through the perspective of models of computation has value. It allows the leveraging of results from theoretical computer science to get an idea of the sort of problems that HCH could solve. In some sense, it’s more OCO -- as in “Oracles Consulting OCO”.

Knowledge about HCH as a model of computations is thus relatively analogous to knowledge for Turing Machines:

  • Upper bounds for computability and complexity (algorithms)
  • Lower bounds for computability and complexity (impossibility results)
  • Requirements or structural constraints to solve a given problem

In summary, this perspective implies the following assumptions about HCH.

  • (Identity) HCH is a model of computation where the human is either a Turing Machine or an oracle satisfying some properties.
  • (Important Questions) These include what can be computed on this model, at which cost, and following which algorithm. But anything that is ordinarily studied in theoretical computer science goes: simulation between this model and others, comparison in expressivity,...
Applications of perspectives on HCH

Armed with our varied perspective on HCH, we can  now put in context different strands of research about HCH that appear incompatible at first glance, and to judge them with the appropriate standards. The point is not that these arguments are correct; we just want to make them clear. After all, it’s not because someone makes sense that they’re right (especially if they’re disagreeing with you).

Here are three examples from recent works, following the three perspectives on HCH from the previous section. Yet keep in mind that most if not all research on this question draws from more than one perspective -- I just point to the most prevalent.

Assumptions about H to have an aligned Question-Answerer

In a recent post (that will probably soon have a followup focused on HCH), Joe Collman introduced the Question-Ignoring Argument (QIA) in Debate:

  • For a consequentialist human judge, the implicit question in debate is always “Which decision would be best for the world according to my values?”.
  • Once the stakes are high enough, the judge will act in response to this implicit question, rather than on specific instructions to pick the best answer to the debate question.
  • With optimal play, the stakes are always high: the value of timely optimal information is huge.
  • The “best for the world” output will usually be unrelated to the question: there’s no reason to expect the most valuable information to be significantly influenced by the question just asked.
  • The judge is likely to be persuaded to decide for the “best for the world” information on the basis of the training signal it sends: we already have the high-value information presented in this debate, even if it doesn’t win. The important consequences concern future answers

The gist is that if the human judge is assumed to have what we would consider outstanding qualities (being a consequentialist that wants to do what’s best for the world), then there is an incentive for the debaters to give an answer to a crucially important question (like a cure for a certain type of cancer) instead of answering the probably very specific question asked. So there is a sense in which the judge having traits we intuitively want makes it harder (and maybe impossible) for the system to be a question-answerer, even if it was the point of the training.

Applying the same reasoning to HCH gives a similar result: if H is either an altruistic human or a model of such a human, it might answer more important questions instead of the one it was asked.

Regardless of whether this line of thinking holds, it provides a very good example of taking HCH as a philosophical abstraction and investigating how much it fits the intuitions for enlightened judgement. Here the intuition is that the enlightened judgement about a question is an enlightened answer to this question, and not just a very important and useful (but probably irrelevant to the question) information.

Experimental work on Factored Cognition

Ought has been exploring Factored Cognition through experiments for years now. For example, their latest report studies the result of an experiment for evaluating claims about movie review by seeing only one step of the argument.

Such work indirectly studies the question of the competitiveness of HCH. In a sense, the different Factored Cognition hypotheses are all about what HCH can do. This is crucial if we aim to align question-answers by means of approximating HCH.

The Ought experiments attempt to build a real-world version of (necessarily bounded) HCH and to see what it can do. They thus place themselves in the perspective of HCH as an intermediary alignment scheme, focusing on how competitive various approximations of it are. Knowing this helps us understand that we shouldn’t judge these experiments through what they say about HCH alignment for example, because their perspective takes it for granted.

HCH as Bounded Reflective Oracle

In Relating HCH and Logical Induction, Abram Demski casts HCH as a Bounded Reflective Oracle (BRO), a type of probabilistic oracle Turing Machine which deals with diagonalization and self-referential issue to answers questions about what an oracle would do when given access to itself (the original idea is of Reflective Oracle -- the post linked above introduces the boundedness and the link with Logical Induction.) This reframing of HCH allows a more formal comparison with Logical Induction, and the different guarantees that they propose.

The lack of consideration of the human makes this post confusing when you think first of HCH as a philosophical abstraction of enlightened judgement, or as an intermediary alignment scheme. Yet when considered through the perspective of HCH as a model of computation, this makes much more sense. The point is to get an understanding of what HCH actually does when it computes, leveraging the tools of theoretical computer science for this purpose.

And once that is clear, the relevance to the other perspectives starts to appear. For example, Abram talks about different guarantees of rationality satisfied in Logical Induction, and why there is no reason to believe that HCH will satisfy them by default. On the other hand, that raises the question of what is the impact of the human on this:

It would be very interesting if some assumptions about the human (EG, the assumption that human deliberation eventually notices and rectifies any efficiently computable Dutch-book of the HCH) coud guarantee trust properties for the combined notion of amplification, along the lines of the self-trust properties of logical induction.


HCH is still complex to study. But I presented multiple perspectives that help clarify most discussions on the subject: as a philosophical abstraction, as an intermediary alignment scheme, and as a model of computation.

Nothing guarantees that these are the only fruitful perspectives on HCH. Moreover, these might be less useful than imagined, or misguided in some ways. Yet I’m convinced, and I hope you’re more open to this idea after reading this post, that explicit thinking about which epistemic perspectives to take on an idea like HCH matters to AI Alignment. This is one way we make sense of our common work, both for pushing more research and for teaching the newcomers to the field.


How would free prediction markets have altered the pandemic?

9 февраля, 2021 - 13:55
Published on February 9, 2021 10:55 AM GMT

In WebMD and the Tragedy of Legible Expertise, Scott Alexander ends by saying that we should be grateful for the experts we have, but that prediction markets would still be better.

Is he right? If so, how much better would prediction markets be? Let's do some counterfactual history. Conditional on large, liquid, free prediction markets being made suddenly available to the public in Nov 2019 (or April 2020, or any other date), what would have been the most likely pandemic outcomes? I'm interested in both vague and specific answers. Even better if you can estimate a quantity of money or life-years saved.


Against butterfly effect

9 февраля, 2021 - 13:43
Published on February 9, 2021 7:46 AM GMT

It is known that, when you simulate a deterministic system described by nonlinear differential equations, small differences in the initial conditions can be exponentially amplified, resulting in huge differences in the final result.

To describe this phenomenon Edward Lorentz famously said that “a butterfly flapping its wings in Brazil can produce a tornado in Texas”. This quote, popularized by Gleick’s 1987 bestseller on chaos theory, came to mean that small events and small decisions can have huge and unpredictable consequences.

The problem with this conception is that it is extrapolating from only two data points a correlation that (almost surely) does not exist. 

Let us suppose that some aliens run a simulation of our universe, starting in 1 January, with our present initial conditions x(0). This simulation could be deterministic or probabilistic, depending on your philosophical standpoint on how our universe works. The aliens go on simulating until 1 July, and observe that on 1 July there is no tornado in Texas. Then they run again the simulation, but this time they slightly modify the initial condition x(0) (a butterfly flips its wings). This time, on 1 July there is a tornado in Texas. Does this couple of observation mean, in any meaningful way, that the butterfly caused the tornado?

To answer this question, we must run many simulations, sampling all the possibile initial conditions. If our universe is not deterministic, it would make sense also to repeat many times the simulation for each initial condition. Then we could measure the correlation between the correlation between the event “butterfly flips wings on 1 january” and the event “tornado in Texas on 1 july”. But this correlation will be almost likely 0. The more the system is chaotic, the more correlations will decay exponentially with time.

There are systems (like human history) in which small decisions can have big consequences. For example, I guess that the aliens simulating our universe could detect some positive correlation between the events “Francis Ferdinand gets shot in 1914” and “New Zealand is at war in 1917”. I do not think that this correlation is very big, but it could be fairly greater than 0. But this is because Francis Ferdinand heir to the throne was a very special person, whose life had big and predictable correlations with the lives of millions of other people.

If the system is predictable, it is easy to think to cases of big correlations between small decisions and big events. Pressing a small button can start a factory. But you can not control weather by waving at the wind. The more the system is caothic, and the more it is exponentially unlikely to have big correlations.


Evolution from distinction difference

9 февраля, 2021 - 08:20
Published on February 9, 2021 5:20 AM GMT

If we have norms such that each copy of a common behavior must to be a tiny step away from from its parent, rather than a giant step or no step, this would seem to make culture much more amenable to gradient descent via evolution than it otherwise would be.

Is the latter somehow reason for us seeing the former? For instance, did ancient groups who frowned on really weird people and who felt awkward being too conformist outflourish other groups with their better evolved cultural norms and artifacts?

Also, is this picture true and applicable to the real world? Is the size of these steps in human culture such that culture learns well? Can you see such effects in the world?

Is this why women’s clothing—which seems expected to vary more between women—also changes faster over time than men’s clothing? (Is that even true?)


Promoting Prediction Markets With Meaningless Internet-Point Badges

8 февраля, 2021 - 22:08
Published on February 8, 2021 7:03 PM GMT



I'd like to live in a world where prediction-market use is common and high-prestige.

The easiest way for this to happen is for prediction markets with money to be legal.

In the absence of this, there might nevertheless be some potential low-hanging fruit for a point-based prediction market -- Metaculus, or some unidentified contender -- to promote the wide acceptance of prediction markets. The same action might also improve the general quality of journalism, potentially.


The prediction market creates a new feature. The feature allows a user of the market to create a small badge, displayable on the user's blog, Medium, Substack, or elsewhere, that displays the person's username and a score measuring the accuracy of their predictions.

The score could be an absolute measure such as Brier score, or a relative measure such as the percentile that the person occupies within the market. It could also be colored according to the number of predictions the person has made; or it could have a tag indicating that this accuracy only obtains within a particular subject or field; or it could indicate the time horizon with which they typically make predictions; generally, there are numerous addendums that could be added.

All of the above details are important, but for the moment I put them to the side.

The badge could be displayed at the head of every article by the author, potentially.

Codewise, this would work similarly to any front-end widget managed by another server, i.e., like a commenting system, like a Twitter embed, and so on and so forth. So of course it would update live as the predictions by the author came true / did not come true, even on older articles.

(This badge could be supplemented, of course, by embedded questions from prediction markets, which could be placed in articles. Metaculus already has these.)

Possible Points in Favor

People are tired of shitty media. There's an enormous groundswell of media distrust from many angles, as far as I can tell. A measure like this is easy to understand, at least in the basics, and provides clear evidence of credibility for those who use it, entirely independent of trust.

It also evens the credibility playing-field between individuals and large agencies, which could be popular.

People like little badges if they grant status. If the first users of this are sufficiently high-prestige, or if predictions / articles made by users of this badge gain fame, then many people will want this. (After all, people wanted to get the Twitter verified badge, right?) This could lead more people to the prediction market, which would be good.

Tying narrative to numbers helps broad acceptance of prediction markets. Prediction markets are great, but prediction markets are not stories, and people love stories. Having people write journalist-y narratives within the context of their personal predictions could then make prediction markets more popular, while also constraining said people to attend more carefully to the truth.

Possible Points Against

Writers don't want auditability. This is true; a lot of writers do not. If enough writers start using this, though, ideally the lack of such a badge would be considered strong evidence the writer does not take truth seriously, and it would therefore become in the interests of writers to include it.

People just won't start using it. I think the most difficult part, here, is getting an initial quantity of writers to start using such a badge. A prediction market could help this by enlisting some famous people to start using it. But I freely admit early acceptance is the trickiest part. I'm not sure what the best approach is.

There's a host of generic objections that also apply equally well to all prediction markets, which I will not here address.

Honestly, not sure if this would work or not.  But I think there's a possible world where it could help a lot.


What metrics should I track when experimenting with sleep?

8 февраля, 2021 - 20:22
Published on February 8, 2021 5:22 PM GMT

I used to sleep very little (6 hours or so) and function well with extremely good academic achievement. Instead of sleeping 8 hours now, I would love those extra 2 hours each day again but I'm not sure it won't affect my cognitive ability in the short term. I don't trust my impressions and would love to have some quantitative data.

  • What are areas that might be affected the most (e.g., long-term memory, working memory, speed of arithmetic calculations, vocabulary, reasoning, mood, etc.)?
  • What are some tasks that would measure my performance in those areas? Ideally those tasks should be:
    • quick to perform
    • easy to implement (I could write a quick program to run those but if there is a web app somewhere that does that already, that would be ideal)
    • fine with the subject knowing what is being measured (I'm going to apply them to myself, after all)
    • produce similar results regardless of how much experience I have with them (I don't want to wonder whether improvement/regression is due to the changes in my sleep or getting better/tired with the task.) It's ok if the skill builds over a week or so and then stays constant.

 I'm not attempting to prevent long-term negative consequences since AFAICT nobody knows what those might be. Besides, there are significant changes in my lifestyle every month or so and so there would be too much noise in the data.


Time to Count or Count to Time?

8 февраля, 2021 - 19:27
Published on February 8, 2021 4:27 PM GMT

Epistemological Status: There's been a fair amount of discussion regarding counting over the last couple of weeks. I'm not aiming to be correct here, just less wrong than when I started thinking about this.

Tell Me In a Low Count of Words:

Counting itself is useful insofar as it allows an entity to order a succession of recognitions. The most basic recognition is perception. To recognize the succession of perceptions is an internal abstraction of time. I'll argue that this comes before everything else. Whenever the succession of recognitions does not depend on the contents of the recognition, just that they occur, counting is a useful abstraction. There is also a possibility of relating external abstractions for counting to the internal ones which creates an interesting coordination problem / numbers.

Tell Me in a Long Count of Words:

I think it's important to start by noting that counting is hard. AllAmericanBreakfast recounts a disagreement for an inventory count on a project to install signage for a college campus's botanical collection. However, no one could agree on how many posts were installed.

They spent a significant amount of time pinning down an 'exact' number in order to create consensus around where their project stands. Reflecting on this they write,

...[W]e should be relieved that the project of "getting hard data" (i.e. science) is able to create some consensus some of the time....Strategically, the "hardness" of a number is its ability to convince the people you want to convince, and drive the work in the direction you think it should go.

Numbers allow for synchronization or coordination. Eigil Rischel also thinks counting is hard. They note that in experiments grown human brains don't come hardwired with an arbitrarily powerful "compare the size of two collections" module. Counting is a technology - it had to be invented.


Given the hardness of counting, can we at least recognize when it would be useful? Johnswentworth proposes something abstract here. Take the inventory disagreement. There is job to be done. Let's pose the following query that takes in the state and returns an answer to, q1:What needs to be done to finish?.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} AllAmericanBreakfast looks at the state of the project and wants to make a determination about what should be done next. Say the state loosely consisted of an abstraction of the botanical garden as it is. The goal is explicitly to install signs so it seems clear we should pose the following query that takes in an (location) attention distribution and the map and returns, q2:Is there a sign here? Signs occupy space, so there can only be so many non-overlapping locations signs could be. Therefore the number of locations is finite and we can form the query collection, Q:Return an inventory of the signs AllAmericanBreakfast specifically writes that,

In our meeting with the VP yesterday, this inventory helped cut through the BS and create a consensus around where the project stands. Instead of arguing over whose estimate is correct, I can redirect conversation to "we need to get this inventory done so that we'll really know."

In other words, we should be able to do the reduction, q1(s)→q1(Q(s)) Now an inventory of the signs would be something like a collection or set of locations for the current signs. Say we swapped the location of two of the signs does that change the final estimate we're interested in? No. This is where Johnswentworth's idea comes into play,

Formally, we can define a function COUNTS such that,

  1. A function for which any f(x,y,z,…) invariant under reordering inputs can be written as f(x,y,z,…)=g(COUNTS(x,y,z,…)) for some g.
  2. COUNTS is invariant under reordering its inputs

According to this definition if the reduction above holds what remains to be doesn't change if we swap the location of two signs then we have further reduction, q1(s)→q1(Q(s))→q3(COUNTS(Q(s))) In words, if we count the number of signs in the botanical then we can make a determination of what needs to be done to finish. To whatever degree what's left to be done only depends on the result of COUNTS is the degree to which the last reduction is valid.

Structural Reductions

One problem with the previous argument is that the problem of modeling the inventory via Q is non-trivial. We've replaced the problem of recognizing numbers with the problem of recognizing (sets) abstraction.

One alternative is to use recursion. However, John argues that most applications of numbers do not involve induction in any obvious way - not even implicitly. On the other hand, AllAmericanBreakfast and Eigil Rischel seem to have two halves of the argument I'm about the present.

Eigil notes that there is a simple technology available to count,

...pair off the elements one after the other. If the collections are exhausted at the same time, they're the same size.

AllAmericanBreakfast suggests that "getting hard data" is a social coordination mechanism. We have to agree on our abstractions.

It seems relatively uncontroversial to note that we can determine the sucession of events. We can ask the question: does this event succeed the other? Yet, this requires something. Through whatever means we perceive the world there must also be an additional abstraction that attaches itself to each perception. It must also be the case that the succeeding perception is attached with an abstraction that can be surely determined as succeeding.

So to each event there is a successor. This sounds like counting, but considerations of our intuition for time tend to get into hairy philosophical issues so I'll take a different route to finish this argument.

Abstract Nonsense:

The point can be made symbolically. Say we combine observations ot together with internal abstractions qt into new abstractions with an abstraction operator A, qt+1=A(qt,ot) If A was something like a RNN this would be Turing complete. Note that to even discuss this we need to index the succession of internal abstractions. Not everything happens at once. What we want is for the query h, h(qi,qj)⟺(qi≺qj)⟺(i≺j) to be learnable. As we noted above, the bare minimum is a mapping, t(qi)∈(S,≤) into an ordered set. We also know that each event has an immediate successor. This kind of rules out a lot of bizarre looking options. We might as well just take S≅N or continue along with our requirements, t(qt)<t(A(qt))=t∘A(qt)=succ∘t(qt) Abstractly, our requirement is that the abstraction operator be conjugate to the successor relation. This is where the fun is. Vaguely we want something like this, (qt→Aqt+1)⇒(t(qt)→succt(qt)+1) This is a map that abstracts the abstraction operator. If we're not careful, this will become trivial, if it isn't already. Instead look at the following diagram,

We want to learn a single map τ that makes everything (commute) work out. The abstraction operator is the only real free 'variable' in this diagram. If we want to determine the succession internal abstractions A must be constrained to fit into the above picture.


The above argument isn't about showing that every human already has 'numbers' floating around in their head, the point is that numbers can be abstracted out of every human's head. Subtle, but different. The significance is that it's easier to modify a pre-existing counting structure than to create one from scratch.

In fact, there are ways to create new external abstractions based on our internal abstractions. Maybe we make tallies, use a sundial, or place pebbles in such a way that we can learn a mapping between pebbles and our internal abstraction for succession. However, this ability to externalize an internalize abstraction is a serious piece of technology.

The issue arises when there are multiple people because then there are multiple internal abstractions. A lot of coordination needs to happen in order for a single external abstraction to map well onto the group's internal abstractions. A lot of the innovation lies in creating a good external abstraction. A lot of the work lies in teaching individuals how to relate them to their internal abstractions.


  1. Is 'tracking' time more fundamental than 'corresponding' objects when it comes to counting?

  2. Are external abstractions 'real' or are they just an internal abstraction that can be shared?

  3. Does coordinating the external representation of time actually lead to counting?


Were the Great Tragedies of History “Mere Ripples”?

8 февраля, 2021 - 18:35
Published on February 8, 2021 3:35 PM GMT

It's been a year, but I finally wrote up my critique of "longtermism" (of the Bostrom / Toby Ord variety) in some detail. I explain why this ideology could be extremely dangerous -- a claim that, it seems, some others in the community have picked up on recently (which is very encouraging). The book is on Medium here and PDF/EPUB versions can be downloaded here.


The distinction distance

8 февраля, 2021 - 10:40
Published on February 8, 2021 7:40 AM GMT

People have a strong tendency to be different from one another (e.g. are horrified to be caught in the same dress, find it weird to order the same dishes as their companion without comment or to choose the same art for their living room). Yet they also have a strong tendency to conform.

These are even in the same areas, and the best behavior seems to be balancing on an edge between the two forces. You don’t want to wear either a dress that someone else is wearing, nor a dress in a style that hasn’t been worn since the 1600s.

I have noticed both of these human forces before, but I hadn’t seen them so vividly as acting in the same realm. You don’t want your essay to be on an identical topic to another student’s, but you also don’t want it to be outside the bounds of what the professor thinks of as an essay, or expressing views beyond a short hop from those others would endorse.

This makes me imagine the curlicues of culture as growing in the fertile interstitial zone between options too conformist to consider and options too wild to consider. Kind of like a Mandelbrot set or a tidal flat or a cellular automaton. There’s a similar pattern in the space of ways the whole of culture could have been: if everyone was very conformist about everything, it would be monotony, and if everyone immediately flung themselves as far away from anyone else as they could on every axis, it would be another kind of monotony. But with this balance of effects as it is, we get some complicated spiraling evolution of art movements and attitudes, trousers and tools. Each idea bringing forth riffs of of it in every direction.

Inspired by a conversation with Robin Hanson, where he probably basically said the main point here, that these two forces act in opposition.


Quadratic, not logarithmic

8 февраля, 2021 - 07:04
Published on February 8, 2021 3:42 AM GMT

I recently realized a very simple, almost obvious bias, that I had because I never thought more about it. Moreover, quite a lot of people have this bias too.

What is worse in time of pandemic - to increase the number of your contacts from 0 to 1 or from 99 to 100? Intuitively, since we perceive many quantities on a logarithmic scale, the first thing is much worse. I heard multiple times something like: "I am already doing this and this and that because I have to, so it does not make sense for me to decrease my shopping", or "My roommate (spouse, child...) does not care about this at all, so it does not make sense for me either". 

However, this is simply not true. If I care solely about myself, increasing the number of contacts increases the probability to get sick linearly - no logarithmic scale. But if I also care about other people (my contacts, yes), then we have linear growth of probability to become a spreader, and linear growth of the group to whom I can spread, thus leading to quadratic growth of the total expected damage to society.

So, if I have quite a lot of contacts already, I should be much more cautious about adding more than if I have almost none. It sounds so trivial right now - yet so many times I have heard the opposite advice. 


Metric selection bias: why Moore's law is less important than you think

8 февраля, 2021 - 03:21
Published on February 8, 2021 12:21 AM GMT

1. Intro

Before vaccine-induced-neo-techno-optimism was all the rage, it was fashionable and popular (in some circles) to bemoan our era as one of disturbing technological stagnation. Almost a decade ago, economists like Paul Krugman and Larry Summers started lamenting the secular stagnation in economic growth. More recently, a small cohort have argued pretty convincingly that technological progress - the fundamental driver of growth and improved standard of living - is slowing down.

A few sourced from the first link above:

Just a few days ago, Jason Crawford wrote a nice summary of the evidence. Less rigorous but perhaps more impactful have been the bounty of pithy aphorisms from technologists like Peter Thiel:

  • They promised us flying cars and all we got was 140 characters.
  • You could say that all these gadgets and devices, they dazzle us but they also distract us from the ways in which our larger surroundings are strangely old. So we run cell phones while we’re riding in a 19th-century subway system in New York. San Francisco, the housing stock looks like it’s from the 50s and 60s, it’s mostly quite decrepit and incredibly hard to change these sort of things. So you have bits making progress, atoms are strangely very stuck.
2. The Problem

My purpose here isn’t to debunk the claims that technological progress is slower than it used to be, than it ought to be, or than many think it is. After writing a paper on the topic, I tend to agree with Cowen and Thiel. One thing, though, rubs me the wrong way. Quite often, techno-pessimist papers and articles point to things like a slowdown in Moore’s Law as evidence for technological stagnation.

Don’t take my word for it. From Cowen and Southewood’s paper:

Still, the exact same data used to illustrate Moore’s Law now suggest that Moore’s Law definitely is slowing down. And that is evidence for a scientific slowdown in what arguably has been the world’s single most dynamic sector, namely information processing.

From Are Ideas Getting Harder to Find?:

The number of researchers required today to achieve the famous doubling every two years of the density of computer chips is more than 18 times larger than the number required in the early 1970s.

From Isolated Demands for Rigour in New Optimism:

But wait a minute, Intel is no longer the most Moore’s Law-relevant company. Their 7nm process was delayed to 2022, and they no longer lead the pack.

Instead, TSMC is now one of only two fabs (including Samsung) able to keep up with Moore’s Law. (For what it’s worth, they also manufactured the Apple M1 chip.) This is their R&D data from Bloom, with the last few years added.

What concerns me is that these authors select Moore’s law for analysis because it is a salient demonstration of extraordinary past technological achievement. In other words, it isn’t merely a random draw from the countless plausible metrics of scientific progress (say, cost of shipping one kilogram from New York to L.A. or the proportion of infants who live to 100). There’s a term for this fallacy: selection bias.

3. Why it’s misleading

If we look at the progress within various scientific fields and industries over time, there will be plenty of variation; in any given decade, some fields will dramatically improve their understanding of the world - perhaps through something like one of Kuhn’s Scientific Revolutions - while others toil away for slow, marginal advances. More dramatically, some disciplines come into existence (computer science), while others fizzle away as their fundamental assumptions are exposed as useless or incorrect (alchemy, astrology).

However, successes and disappointments are unlikely to be equally salient. People remember how their lives were changed by the radio, automobile, or smartphone, but don’t automatically pay attention to the science and technology that aren’t causing much change.

That’s why so much of the techno-pessimist literature has to use thought experiments (“Go into a room and subtract off all of the screens. How do you know you’re not in 1973, but for issues of design?” from Crawford’s “Technological stagnation”) and the like to remind us of all the progress that isn’t happening. People naturally notice Facebook, Uber, AlphaGo, and rapid vaccine production, but have to be explicitly reminded that things like physical infrastructure and transportation are largely no better than they were a few decades ago.

This asymmetry has a few interesting consequences. First, it likely causes us to intuitively overestimate current technological progress before we do the more systematic analyses like those I’ve cited. Second, it means that more rigorous and systematic analyses, which tend to compare the rates of change in various metrics between past and present, are prone to do exactly the opposite.

Here’s Why

Any comparison of current to past technological progress is likely to select the most salient, well-known indicators of scientific and technological progress. For the reason I just described, these indicators are likely to be associated with the most successful, rapidly-advancing fields and industries.

There’s no better example than Moore’s Law. In the last 50 years or so, the (literally) exponential rise in transistor density has enabled computation to transition from an academic novelty to a core component of virtually every facet of modern life.

The timing here isn’t coincidental; the most successful and influential academics and entrepreneurs like Cowen (59) and Thiel (53) grew up with Moore’s Law and all the progress it represents in full effect.

Of course, this sort of selection bias sets you up for disappointment thanks to regression to the mean. Whether we’re talking about NFL teams, mutual funds, or scientific fields, the most successful units today are probably going to look a little more average tomorrow.

The cost of solar power has been plummeting recently, making it cost-competitive with fossil fuels. As you can see, though, this is not representative of other forms of energy - including climate-friendly ones like wind and nuclear.

I’m not sure whether this trend has a name yet, but let’s call it Noore’s Law. In 40 years, the next generation of economists might ruefully point to the flatlining cost of solar energy as evidence for technological stagnation. It’s not that this is incorrect - it’s that failing to consider it in the context of other, more normal metrics (like the costs of wind and nuclear energy) assigns Noore’s Law undue importance.

4. Not Just Transistors

I’ve used Moore’s Law to illustrate my point so far because it is such a perfect example of a metric pre-selected to show technological slowdown. Tons of authors refer to it, its timing is perfect, and it clearly represents an especially impactful and successful past industry.

That said, there are plenty of other examples. Both “Is the rate of scientific progress slowing down?” and “Are Ideas Getting Harder to Find,” for instance, find that crop yield (such as bushels of wheat per acre) growth rates are declining, even though the number of researchers involved is growing.

Agricultural productivity has been a central concern of human civilization for thousands of years, so this does seem a less arbitrary choice than Moore’s Law. Nonetheless, it seems very likely to me that such a metric appears to be a good measure of scientific progress precisely because crop yields have increased so dramatically over the last few hundred years.

I doubt this is nefarious or intentional distortion. Instead, we simply think of those things which have dramatically increased in the recent past as the kind of thing that are supposed to keep increasing.

And an exception to the rule

Let’s take another example that, while indeed selected for past success in this way, is nonetheless evidence for a technological resurgence: vaccines.

From Noah Smith’s article

Vaccines are perhaps the most dramatic medical success of recent history. Diseases that used to kill thousands of people have become a thing of the past, all thanks to an extremely cheap and widely-available technology. So, rates of communicable disease, direct rates of vaccination, or a more qualitative assessment of vaccine development speed/impressiveness/quality are all pretty salient examples of past technological achievement.

Ex ante, we should expect analyzing these metrics over time to be biased toward showing technological slowdown. However, regression to the mean is a tendency, not a law. Sometimes, data points that are already far from the mean at time t get even further further at t+1.

Vaccines are a case in point. Matt Yglesias, in “Some optimism about America's Covid response,” summarizes the neo-techno-optimist take:

And what’s particularly great about these new vaccines is they’re the fastest we’ve ever seen developed (the previous record was four to five years), and they’re based on a whole new kind of vaccine technology. So we’re not only getting new vaccines but new ways to make vaccines, which is really cool.

I’m nowhere near knowledgeable enough to assess whether the recent COVID vaccines indeed represent significant progress, but I’ll take Yglesias at his word. Even though vaccination seems to show evidence in the direction opposite that of Moore’s Law and agricultural productivity, all three are selected for analysis (and thus biased) because they show past technological progress.

5. The mandatory concession paragraph

None of this is to say that we shouldn’t look at things like Moore’s law. These metrics are misleading when decontextualized for precisely the same reason they have and continue to be important for society. Likewise, using them as evidence weakens but does not negate one’s argument. In fact, I tend to agree with the techno-pessimists that tech progress is slower than in the 19th and 20th centuries.

Nonetheless, a good evaluation of the state of science and tech needs to account for the selection bias I’ve described in order to draw more rigorous, robust, and accurate conclusions about the state of our society.