Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 25 минут 5 секунд назад

Focusmate: Cryonics sign up

1 марта, 2021 - 18:41
Published on March 1, 2021 3:41 PM GMT

Event will be here: https://meet.google.com/ari-sqxa-qnh

Focusmate is an event format where we all work at the same time, but without talking. It can help get things done.

During this time we will all work on making progress towards:

  • deciding whether we want to sign up for cryonics
  • making a plan for when to sign up for cryonics (if not now)
  • making progress towards being signed up for cryonics

I'm already signed up for cryonics, but I will be here to answer people's questions by text. Why by text?

  • Because I will document the answers one the Biostasis Wiki
  • To avoid distracting others

If more than 0 people attend, I will organize another one the first Sunday of next month.


Fun with +12 OOMs of Compute

1 марта, 2021 - 16:30
Published on March 1, 2021 1:30 PM GMT

Or: Big Timelines Crux Operationalized

What fun things could one build with +12 orders of magnitude of compute? By ‘fun’ I mean ‘powerful.’ This hypothetical is highly relevant to AI timelines, for reasons I’ll explain later.

Summary (Spoilers):

I describe a hypothetical scenario that concretizes the question “what could be built with 2020’s algorithms/ideas/etc. but a trillion times more compute?”  Then I give some answers to that question. Then I ask: How likely is it that some sort of TAI would happen in this scenario? This second question is a useful operationalization of the (IMO) most important, most-commonly-discussed timelines crux:  “Can we get TAI just by throwing more compute at the problem?” I consider this operationalization to be the main contribution of this post; it directly plugs into Ajeya’s timelines model and is quantitatively more cruxy than anything else I know of. The secondary contribution of this post is my set of answers to the first question: They serve as intuition pumps for my answer to the second, which strongly supports my views on timelines.

The hypothetical

In 2016 the Compute Fairy visits Earth and bestows a blessing: Computers are magically 12 orders of magnitude faster! Over the next five years, what happens? The Deep Learning AI Boom still happens, only much crazier: Instead of making AlphaStar for 10^23 floating point operations, DeepMind makes something for 10^35. Instead of making GPT-3 for 10^23 FLOPs, OpenAI makes something for 10^35. Instead of industry and academia making a cornucopia of things for 10^20 FLOPs or so, they make a cornucopia of things for 10^32 FLOPs or so. When random grad students and hackers spin up neural nets on their laptops, they have a trillion times more compute to work with.

For context on how big a deal +12 OOMs is, consider the graph below, from ARK. It’s measuring petaflop-days, which are about 10^20 FLOP each. So 10^35 FLOP is 1e+15 on this graph. GPT-3 and AlphaStar are not on this graph, but if they were they would be in the very top-right corner.

Question One: In this hypothetical, what sorts of things could AI projects build?

I encourage you to stop reading, set a five-minute timer, and think about fun things that could be built in this scenario. I’d love it if you wrote up your answers in the comments!

My tentative answers:

Below are my answers, listed in rough order of how ‘fun’ they seem to me. I’m not an AI scientist so I expect my answers to overestimate what could be done in some ways, and underestimate in other ways. Imagine that each entry is the best version of itself, since it is built by experts (who have experience with smaller-scale versions) rather than by me.


In our timeline, it cost about 10^23 FLOP to train AlphaStar. (OpenAI Five, which is in some ways more impressive, took less!) Let’s make OmegaStar like AlphaStar only +7 OOMs bigger: the size of a human brain.[1] Larger models seem to take less data to reach the same level of performance, so it would probably take at most 10^30 FLOP to reach the same level of Starcraft performance as AlphaStar, and indeed we should expect it to be qualitatively better.[2] So let’s do that, but also train it on lots of other games too.[3] There are 30,000 games in the Steam Library. We train OmegaStar long enough that it has as much time on each game as AlphaStar had on Starcraft. With a brain so big, maybe it’ll start to do some transfer learning, acquiring generalizeable skills that work across many of the games instead of learning a separate policy for each game.

OK, that uses up 10^34 FLOP—a mere 10% of our budget. With the remainder, let’s add some more stuff to its training regime. For example, maybe we also make it read the entire internet and play the “Predict the next word you are about to read!” game. Also the “Predict the covered-up word” and “predict the covered-up piece of an image” and “predict later bits of the video” games.

OK, that probably still wouldn’t be enough to use up our compute budget. A Transformer that was the size of the human brain would only need 10^30 FLOP to get to human level at the the predict-the-next-word game according to Gwern, and while OmegaStar isn’t a transformer, we have 10^34 FLOP available.[4] (What a curious coincidence, that human-level performance is reached right when the AI is human-brain-sized! Not according to Shorty.)

Let’s also hook up OmegaStar to an online chatbot interface, so that billions of people can talk to it and play games with it. We can have it play the game “Maximize user engagement!”

...we probably still haven’t used up our whole budget, but I’m out of ideas for now.


Let’s start by training GPT-7, a transformer with 10^17 parameters and 10^17 data points, on the entire world’s library of video, audio, and text. This is almost 6 OOMs more params and almost 6 OOMs more training time than GPT-3. Note that a mere +4 OOMs of params and training time is predicted to reach near-optimal performance at text prediction and all the tasks thrown at GPT-3 in the original paper; so this GPT-7 would be superhuman at all those things, and also at the analogous video and audio and mixed-modality tasks.[5] Quantitatively, the gap between GPT-7 and GPT-3 is about twice as large as the gap between GPT-3 and GPT-1, (about 25% the loss GPT-3 had, which was about 50% the loss GPT-1 had) so try to imagine a qualitative improvement twice as big also. And that’s not to mention the possible benefits of multimodal data representations.[6]

We aren’t finished! This only uses up 10^34 of our compute. Next, we let the public use prompt programming to make a giant library of GPT-7 functions, like the stuff demoed here and like the stuff being built here, only much better because it’s GPT-7 instead of GPT-3. Some examples:

  • Decompose a vague question into concrete subquestions
  • Generate a plan to achieve a goal given a context
  • Given a list of options, pick the one that seems most plausible / likely to work / likely to be the sort of thing Jesus would say / [insert your own evaluation criteria here]
  • Given some text, give a score from 0 to 10 for how accurate / offensive / likely-to-be-written-by-a-dissident / [insert your own evaluation criteria here] the text is.

And of course the library also contains functions like “google search” and “Given webpage, click on X” (remember, GPT-7 is multimodal, it can input and output video, parsing webpages is easy). It also has functions like “Spin off a new version of GPT-7 and fine-tune it on the following data.” Then we fine-tune GPT-7 on the library so that it knows how to use those functions, and even write new ones. (Even GPT-3 can do basic programming, remember. GPT-7 is much better.)

We still aren’t finished! Next, we embed GPT-7 in an amplification scheme — a “chinese-room bureaucracy” of calls to GPT-7. The basic idea is to have functions that break down tasks into sub-tasks, functions that do those sub-tasks, and functions that combine the results of the sub-tasks into a result for the task. For example, a fact-checking function might start by dividing up the text into paragraphs, and then extract factual claims from each paragraph, and then generate google queries designed to fact-check each claim, and then compare the search results with the claim to see whether it is contradicted or confirmed, etc. And an article-writing function might call the fact-checking function as one of the intermediary steps. By combining more and more functions into larger and larger bureaucracies, more and more sophisticated behaviors can be achieved. And by fine-tuning GPT-7 on examples of this sort of thing, we can get it to understand how it works, so that we can write GPT-7 functions in which GPT-7 chooses which other functions to call. Heck, we could even have GPT-7 try writing its own functions! [7]

The ultimate chinese-room bureaucracy would be an agent in its own right, running a continual OODA loop of taking in new data, distilling it into notes-to-future-self and new-data-to-fine-tune-on, making plans and sub-plans, and executing them. Perhaps it has a text file describing its goal/values that it passes along as a note-to-self — a “bureaucracy mission statement.”

Are we done yet? No! Since it “only” has 10^17 parameters, and uses about six FLOP per parameter per token, we have almost 18 orders of magnitude of compute left to work with.[8] So let’s give our GPT-7 uber-bureaucracy an internet connection and run it for 100,000,000 function-calls (if we think of each call as a subjective second, that’s about 3 subjective years). Actually, let’s generate 50,000 different uber-bureaucracies and run them all for that long. And then let’s evaluate their performance and reproduce the ones that did best, and repeat. We could do 50,000 generations of this sort of artificial evolution, for a total of about 10^35 FLOP.[9]

Note that we could do all this amplification-and-evolution stuff with OmegaStar in place of GPT-7.

Crystal Nights:

(The name comes from an excellent short story.)

Maybe we think we are missing something fundamental, some unknown unknown, some special sauce that is necessary for true intelligence that humans have and our current artificial neural net designs won’t have even if scaled up +12 OOMs. OK, so let’s search for it. We set out to recapitulate evolution.

We make a planet-sized virtual world with detailed and realistic physics and graphics. OK, not perfectly realistic, but much better than any video game currently on the market! Then, we seed it with a bunch of primitive life-forms, with a massive variety of initial mental and physical architectures. Perhaps they have a sort of virtual genome, a library of code used to construct their bodies and minds, with modular pieces that get exchanged via sexual reproduction (for those who are into that sort of thing). Then we let it run, for a billion in-game years if necessary!

Alas, Ajeya estimates it would take about 10^41 FLOP to do this, whereas we only have 10^35.[10] So we probably need to be a million times more compute-efficient than evolution. But maybe that’s doable. Evolution is pretty dumb, after all.

  1. Instead of starting from scratch, we can can start off with “advanced” creatures, e.g. sexually-reproducing large-brained land creatures. It’s unclear how much this would save but plausibly could be at least one or two orders of magnitude, since Ajeya’s estimate assumes the average creature has a brain about the size of a nematode worm’s brain.[11]
  2. We can grant “magic traits” to the species that encourage intelligence and culture; for example, perhaps they can respawn a number of times after dying, or transfer bits of their trained-neural-net brains to their offspring. At the very least, we should make it metabolically cheap to have big brains; no birth-canal or skull should restrict the number of neurons a species can have! Also maybe it should be easy for species to have neurons that don’t get cancer or break randomly.
  3. We can force things that are bad for the individual but good for the species, e.g. identify that the antler size arms race is silly and nip it in the bud before it gets going. In general, more experimentation/higher mutation rate is probably better for the species than for the individual, and so we could speed up evolution by increasing the mutation rate. We can also identify when a species is trapped in a local optima and take action to get the ball rolling again, whereas evolution would just wait until some climactic event or something shakes things up.
  4. We can optimise for intelligence instead of ability to reproduce, by crafting environments in which intelligence is much more useful than it was at any time in Earth’s history. (For example, the environment can be littered with monoliths that dispense food upon completion of various reasoning puzzles. Perhaps some of these monoliths can teach English too, that’ll probably come in handy later!) Think about how much faster dog breeding is compared to wolves evolving in the wild. Breeding for intelligence should be correspondingly faster than waiting for it to evolve.
  5. There are probably additional things I haven’t thought of that would totally be thought of, if we had a team of experts building this evolutionary simulation with 2020’s knowledge. I’m a philosopher, not an evolutionary biologist!

What about STEM AI? Let’s do some STEM. You may have seen this now-classic image:

These antennas were designed by an evolutionary search algorithm. Generate a design, simulate it to evaluate predicted performance, tweak & repeat. They flew on a NASA spacecraft fifteen years ago, and were massively more efficient and high-performing than the contractor-designed antennas they replaced. Took less human effort to make, too.[12]

This sort of thing gets a lot more powerful with +12 OOMs. Engineers often use simulations to test designs more cheaply than by building an actual prototype. SpaceX, for example, did this for their Raptor rocket engine. Now imagine that their simulations are significantly more detailed, spending 1,000,000x more compute, and also that they have an evolutionary search component that auto-generates 1,000 variations of each design and iterates for 1,000 generations to find the optimal version of each design for the problem (or even invents new designs from scratch.) And perhaps all of this automated design and tweaking (and even the in-simulation testing) is done more intelligently by a copy of OmegaStar trained on this “game.”

Why would this be a big deal? I’m not sure it would be. But take a look at this list of strategically relevant technologies and events and think about whether Skunkworks being widely available would quickly lead to some of them. For example, given how successful AlphaFold 2 has been, maybe Skunkworks could be useful for designing nanomachines. It could certainly make it a lot easier for various minor nations and non-state entities to build weapons of mass destruction, perhaps resulting in a vulnerable world.


According to page 69 of this report, the Hodgkin-Huxley model of the neuron is the most detailed and realistic (and therefore the most computationally expensive) as of 2008. It costs 1,200,000 FLOP per second per neuron to run. So a human brain (along with relevant parts of the body, in a realistic-physics virtual environment, etc.) could be simulated for about 10^17 FLOP per second.

Now, presumably (a) we don’t have good enough brain scanners as of 2020 to actually reconstruct any particular person’s brain, and (b) even if we did, the Hodgkin-Huxley model might not be detailed enough to fully capture that person’s personality and cognition.[13]

But maybe we can do something ‘fun’ nonetheless: We scan someone’s brain and then create a simulated brain that looks like the scan as much as possible, and then fills in the details in a random but biologically plausible way. Then we run the simulated brain and see what happens. Probably gibberish, but we run it for a simulated year to see whether it gets its act together and learns any interesting behaviors. After all, human children start off with randomly connected neurons too, but they learn.[14]

All of this costs a mere 10^25 FLOP. So we do it repeatedly, using stochastic gradient descent to search through the space of possible variations on this basic setup, tweaking parameters of the simulation, the dynamical rules used to evolve neurons, the initial conditions, etc. We can do 100,000 generations of 100,000 brains-running-for-a-year this way. Maybe we’ll eventually find something intelligent, even if it lacks the memories and personality of the original scanned human.

Question Two: In this hypothetical, what’s the probability that TAI appears by end of 2020?

The first question was my way of operationalizing “what could be built with 2020’s algorithms/ideas/etc. but a trillion times more compute?”

This second question is my way of operationalizing “what’s the probability that the amount of computation it would take to train a transformative model using 2020’s algorithms/ideas/etc. is 10^35 FLOP or less?”

(Please ignore thoughts like “But maybe all this extra compute will make people take AI safety more seriously” and “But they wouldn’t have incentives to develop modern parallelization algorithms if they had computers so fast” and “but maybe the presence of the Compute Fairy will make them believe the simulation hypothesis?” since they run counter to the spirit of the thought experiment.)

Remember, the definition of Transformative AI is “AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution.”

Elicit Prediction (elicit.org/binary/questions/lLflClP3r)

Did you read those answers to Question One, visualize them and other similarly crazy things that would be going on in this hypothetical scenario, and think “Eh, IDK if that would be enough, I’m 50-50 on this. Seems plausible TAI will be achieved in this scenario but seems equally plausible it wouldn’t be.”

No! … Well, maybe you do, but speaking for myself, I don’t have that reaction.

When I visualize this scenario, I’m like “Holyshit all five of these distinct research programs seem like they would probably produce something transformative within five years and perhaps even immediately, and there are probably more research programs I haven’t thought of!”

My answer is 90%. The reason it isn’t higher is that I’m trying to be epistemically humble and cautious, account for unknown unknowns, defer to the judgment of others, etc. If I just went with my inside view, the number would be 99%. This is because I can’t articulate any not-totally-implausible possibility in which OmegaStar, Amp(GPT-7), Crystal Nights, Skunkworks, and Neuromorph and more don’t lead to transformative AI within five years. All I can think of is things like “Maybe transformative AI requires some super-special mental structure which can only be found by massive blind search, so massive that the Crystal Nights program can’t find it…” I’m very interested to hear what people whose inside-view answer to Question Two is <90% have in mind for the remaining 10%+. I expect I’m just not modelling their views well and that after hearing more I’ll be able to imagine some not-totally-implausible no-TAI possibilities. My inside view is obviously overconfident. Hence my answer of 90%.

Poll: What is your inside-view answer to Question Two, i.e. your answer without taking into account meta-level concerns like peer disagreement, unknown unknowns, biases, etc.

Elicit Prediction (elicit.org/binary/questions/KohTZ3R3x)

Bonus: I’ve argued elsewhere that what we really care about, when thinking about AI timelines, is AI-induced points of no return. I think this is likely to be within a few years of TAI, and my answer to this question is basically the same as my answer to the TAI version, but just in case:

Elicit Prediction (elicit.org/binary/questions/cy6-bl-zq) OK, here’s why all this matters

Ajeya Cotra’s excellent timelines forecasting model is built around a probability distribution over “the amount of computation it would take to train a transformative model if we had to do it using only current knowledge.”[15] (pt1p25) Most of the work goes into constructing that probability distribution; once that’s done, she models how compute costs decrease, willingness-to-spend increases, and new ideas/insights/algorithms are added over time, to get her final forecast.

One of the great things about the model is that it’s interactive; you can input your own probability distribution and see what the implications are for timelines. This is good because there’s a lot of room for subjective judgment and intuition when it comes to making the probability distribution.

What I’ve done in this post is present an intuition pump, a thought experiment that might elicit in the reader (as it does in me) the sense that _the probability distribution should have the bulk of its mass by the 10^35 mark. _

Ajeya’s best-guess distribution has the 10^35 mark as its median, roughly. As far as I can tell, this corresponds to answering “50%” to Question Two.[16]

If that’s also your reaction, fair enough. But insofar as your reaction is closer to mine, you should have shorter timelines than Ajeya did when she wrote the report.

There are lots of minor nitpicks I have with Ajeya’s report, but I’m not talking about them; instead, I wrote this, which is a lot more subjective and hand-wavy. I made this choice because the minor nitpicks don’t ultimately influence the answer very much, whereas this more subjective disagreement is a pretty big crux.[17] Suppose your answer to Question 2 is 80%. Well, that means your distribution should have 80% by the 10^35 mark compared to Ajeya’s 50%, and that means that your median should be roughly 10 years earlier than hers, all else equal: 2040-ish rather than 2050-ish.[18]

I hope this post helps focus the general discussion about timelines. As far as I can tell, the biggest crux for most people is something like “Can we get TAI just by throwing more compute at the problem?” Now, obviously we can get TAI just by throwing more compute at the problem, there are theorems about how neural nets are universal function approximators etc., and we can always do architecture search to find the right architectures. So the crux is really about whether we can get TAI just by throwing a large but not too large amount of compute at the problem… and I propose we operationalize “large but not too large” as “10^35 FLOP or less.”[19] I’d like to hear people with long timelines explain why OmegaStar, Amp(GPT-7), Crystal Nights, SkunkWorks, and Neuromorph wouldn’t be transformative (or more generally, wouldn’t cause an AI-induced PONR). I’d rest easier at night if I had some hope along those lines.

This is part of my larger investigation into timelines commissioned by CLR. Many thanks to Tegan McCaslin, Lukas Finnveden, Anthony DiGiovanni, Connor Leahy, and Carl Shulman for comments on drafts. Kudos to Connor for pointing out the Skunkworks and Neuromorph ideas.

  1. AlphaStar was 10^8 parameters, ten times smaller than a honeybee brain. I think this puts its capabilities in perspective. Yes, it seemed to be more of a heuristic-executor than a long-term planner, because it could occasionally be tricked into doing stupid things repeatedly. But the same is true for insects.↩︎
  2. This is definitely true for Transformers (and LSTMs I think?), but it may not be true for whatever architecture AlphaStar uses. In particular some people I talked to worry that the vanishing gradients problem might make bigger RL models like OmegaStar actually worse. However, everyone I talked to agreed with the “probably”-qualified version of this claim. I’m very interested to learn more about this.↩︎
  3. To avoid catastrophic forgetting, let’s train OmegaStar on all these different games simultaneously, e.g. it plays game A for a short period, then plays game B, then C, etc. and loops back to game A only much later.↩︎
  4. Lukas Finnveden points out that Gwern’s extrapolation is pretty weird. Quoting Lukas: “Gwern takes GPT-3's current performance on lambada; assumes that the loss will fall as fast as it does on "predict-the-next-word" (despite the fact that the lambada loss is currently falling much faster!) and extrapolates current performance (without adjusting for the expected change in scaling law after the crossover point) until the point where the AI is as good as humans (and btw we don't have a source for the stated human performance) … I'd endorse a summary more like “If progress carries on as it has so far, we might just need ~1e27 FLOP to get to mturk-level of errors on the benchmarks closest to GPT-3's native predict-the-next-word game. Even if progress on these benchmarks slowed down and improved at the same rate as GPT-3's generic word-prediction abilities, we'd expect it to happen at ~1e30 FLOP for the lambada benchmark."
    All that being said, Lukas’ own extrapolation seems to confirm the general impression that GPT’s performance will reach human-level around the same time its size reaches brain-size: “Given that Cotra’s model’s median number of parameters is close to my best guess of where near-optimal performance is achieved, the extrapolations do not contradict the model’s estimates, and constitute some evidence for the median being roughly right.”↩︎
  5. One might worry that the original paper had a biased sample of tasks. I do in fact worry about this. However, this paper tests GPT-3 on a sample of actual standardized tests used for admission to colleges, grad schools, etc. and GPT-3 exhibits similar performance (around 50% correct), and also shows radical improvement over smaller versions of GPT.↩︎
  6. In theory (and maybe in practice too, given how well the new pre-training paradigm is working? See also e.g. this paper) it should be easier for the model to generalize and understand concepts since it sees images and videos and hears sounds to go along with the text.↩︎
  7. GPT-3 has already been used to write its own prompts, sorta. See this paper and look for “metaprompt.” Also, this paper demonstrates the use of stochastic gradient descent on prompts to evolve them into better versions.↩︎
  8. Thanks to Connor Leahy for finding this source for me.↩︎
  9. 50,000 x 50,000 x 100,000,000 x 10^17 x 6 = 1.5x10^35↩︎
  10. Bostrom and Shulman have an earlier estimate with wide error bars: 10^38 - 10^51 FLOP. See page 6 and multiply their FLOPS range by the number of seconds in a year.↩︎
  11. Well, we’d definitely start with small brains and scale up, but we’d make sure to spend only a fraction of our overall compute on small brains. From the report, page 25 of part 3: "the number of FLOP/s contributed by humans is (~7e9 humans) * (~1e15 FLOP/s / person) = ~7e24. The human population is vastly larger now than it was during most of our evolutionary history, whereas it is likely that the population of animals with tiny nervous systems has stayed similar. This suggests to me that the average ancestor across our entire evolutionary history was likely tiny and performed very few FLOP/s. I will assume that the “average ancestor” performed about as many FLOP/s as a nematode and the “average population size” was ~1e21 individuals alive at a given point in time. This implies that our ancestors were collectively performing ~1e25 FLOP every second on average over the ~1 billion years of evolutionary history."↩︎
  12. See page 4 of this paper. Relevant quote: “Originally the ST5 mission managers had hired a contractor to design and produce an antenna for this mission. Using conventional design practices the contractor produced a quadrifilar helix antenna (QHA). In Fig. 3 we show performance comparisons of our evolved antennas with the conventionally designed QHA on an ST5 mock-up. Since two antennas are used on each spacecraft – one on the top and one on the bottom – it is important to measure the overall gain pattern with two antennas mounted on the spacecraft. With two QHAs 38% efficiency was achieved, using a QHA with an evolved antenna resulted in 80% efficiency, and using two evolved antennas resulted in 93% efficiency. … Since the evolved antenna does not require a phasing circuit, less design and fabrication work is required, and having fewer parts may result in greater reliability. In terms of overall work, the evolved antenna required approximately three person-months to design and fabricate whereas the conventional antenna required approximately five months. Lastly, the evolved antenna has more uniform coverage in that it has a uniform pattern with only small ripples in the elevations of greatest interest (40◦ − 80◦ ). This allows for reliable performance as the elevation angle relative to the ground changes.”↩︎
  13. So, no Ems, basically. Probably.↩︎
  14. I mean, definitely not completely random. But I said we’d fill in the details in a random-but-biologically-plausible way. And children simply have far too many neurons for genes to say much about how they connect to each other. Whatever unknowns there are about about how the neurons connect, we can make that part of what’s being optimized by our hundred-thousand-generation search process. The size of the search space can’t be that big, because there isn’t that much space in the human genome to encode any super-complicated instructions. I guess at this point we should start talking about Ajeya’s Genome Anchor idea. I admit I’m out of my depth here.↩︎
  15. Since later I talk about how I disagree with Ajeya, I want to make super clear that I really do think her report is excellent. It’s currently the best writing on timelines that I know of. When people ask me to explain my timelines, I say “It’s like Ajeya’s, except…”↩︎
  16. I think this because I’ve looked at the probability distribution she gives on page 34 of part 3 of her report and 35 OOMs of floating point operations seems to be the median. I had to do this by measuring with a ruler and doing some eyeballing, so it’s probably not exactly correct, but I’d be surprised if the true answer is more than 55% or less than 45%. As a sanity check, Ajeya has 7 buckets in which her credence is distributed, with 30% in a bucket with median 34.5 OOMs, and 35% in buckets with higher medians, and 35% in buckets with lower medians. (But the lower buckets are closer to 35 than the higher buckets, meaning their tails will be higher than 35 more than the higher bucket’s tails will be lower than 35)↩︎
  17. Example: One nitpick I have is that Ajeya projects that the price of compute for AI will fall more slowly than price-performance Moore’s Law, because said law has faltered recently. I instead think we should probably model this uncertainty, with (say) a 50% chance of Moore continuing and a 50% chance of continued slowdown. But even if it was a 100% chance of Moore continuing, this would only bring forward Ajeya’s median timeline to 2045-ish! (At least, according to my tinkering with her spreadsheet)↩︎
  18. Quick calculation: Suppose we take Ajeya’s best-guess distribution and modify it by lowering the part to the right of 10^35 and raising the part to the left of 10^35, until the 10^35 mark is the 80-percentile mark instead of the 50-percentile mark. And suppose we do this raising and lowering in a “distribution-preserving way,” i.e. the shape of the curve before the 10^35 mark looks exactly the same, it’s just systematically bigger. In other words, we redistribute 30 percentage points of probability mass from above 10^35 to below, in proportion to how the below-10^35 mass is already distributed. Well, in this case, then 60% of the redistributed mass should end up before the old 30% mark. (Because the 30% mark is 60% of the mass prior to the old median, the 10^35 mark.) And 60% of 30 percentage points is 18, so that means +18 points added before the old 30% mark. This makes it the new 48% mark. So the new 48% mark should be right where the old 30% mark is, which is (eyeballing the spreadsheet) a bit after 2040, 10 years sooner. (Ajeya’s best guess median is a bit after 2050.) This is, I think, a rather conservative estimate of how cruxy this disagreement is. First, my answer to Question Two is 0.9, not 0.8, and that’s after trying to be humble and whatnot. Second, this procedure of redistributing probability mass in proportion to how it is already distributed produces an obviously silly outcome, where there is a sharp drop-off in probability at the 10^35 mark. Realistically, if you are convinced that the answer to Question Two is 0.8 (or whatever) then you should think the probability distribution tapers off smoothly, being already somewhat low by the time the 80th-percentile mark at 10^35 is reached. Thus, realistically, someone who mostly agrees with Ajeya but answers Question Two with “0.8” should have somewhat shorter timelines than the median-2040-ish I calculated.↩︎
  19. I recommend trying out different numbers depending on who is in the conversation. For conversations in which everyone assigns high credence to the 10^35 version, it may be more fruitful to debate the 10^29 version, since 10^29 FLOP is when GPT-7 surpasses human level at text prediction (and is also superhuman at the other tests we’ve tried) according to the scaling laws and performance trends, I think. For conversations where everyone has low credence in the 10^35 version, I suggest using the 10^41 version, since 10^41 FLOP is enough to recapitulate evolution without any shortcuts. ↩︎


Full-time AGI Safety!

1 марта, 2021 - 15:42
Published on March 1, 2021 12:42 PM GMT

Hi everyone!

I'm Steve Byrnes, a professional physicist in Boston. Y'all may know me as a regular blogger and commenter on this site. (Example post, Full list.)

Well, out of the blue, an exceptionally generous sponsor has granted me a grant to spend the next year full-time trying to advance AGI safety! (Eternal gratitude to Beth Barnes and the CEA donor lottery program!)

To make a long story short, my research plan is to immerse myself in the latest thinking about how the human brain works (i.e., what algorithm does it run?), and apply those insights to help move forward the discussion on AGI safety and strategy. (And conversely, to also help move forward the discussion of "What if we succeed?" within neuroscience.)

Until now I've been researching and blogging in little bits of time squeezed between work and kids. Not anymore! Starting today, I'm in full-time, all-out effort mode!

I know that this year is going to fly by far too quickly; I’m desperate to make the most of it. One thing that will help is lots of exchange of ideas! You can email me at steven.byrnes@gmail.com if you want to see my research proposal, or discuss topics of interest, or just say hi!   :-)


Progressive avant-gardes

1 марта, 2021 - 14:19
Published on March 1, 2021 11:19 AM GMT

[epistemic status: hot take]

In this post I will try to draw parallels between three very different intellectual communities: string theory, Marxist theory, and postmodernism. What do they have in common? Not very much indeed, but I have the impression that there are some analogies worth noticing.

  • They represent an extrapolation of trends that had a very successful track record at the time these movements were born (for string theory, the importance of symmetry arguments, and the unification between different physical theories; for postmodernism, the overcoming of the idea of absolute truth, and of Western cultural imperialism; for Marxism, the belief in equality between people and the disappearance of class privileges).
  • The early followers of these intellectual movement were apparently certain that their new ideas would be eventually adopted by everyone, and that posterity would have regarded them as the first followers of the next big revolution in the respective fields.
  • The followers of these movements, adding a theoretical foundation to the empirical trends described in the first point of this bullet list, developed a shared understanding of how "progress" should like, and why their intellectual movement was apparently the logical next step.
  • They had a lively internal debate, and developed increasingly advanced ideas that require a lot of dedicated study to be understood. As a consequence, their more elaborate writings ended up appearing incomprehensible, or plainly absurd, to people on the outside of these intellectual movements.
  • The basic premises on which their doctrines were built did not age very well (for sting theory, we now know that supersymmetry is very likely false, at least at "natural" energy scales; for Marxism, the theory of value made great improvements since Marx's time; for postmodernism, this point may be false, I do not know). Nevertheless, many of followers of these movement maintain an high level of certainty that they are correct, and that naive criticisms lack the competence (that is, the years of study) required to understand and judge their theories.

Maybe these similarities are enough to group them in a common category?


Subjectivism and moral authority

1 марта, 2021 - 12:02
Published on March 1, 2021 9:02 AM GMT

(Cross-posted from Hands and Cities. Content warning: descriptions of cruelty)

Lots of people I know think that what you should do depends, ultimately, on your contingent patterns of care and concern (suitably idealized), as opposed to some sort of objective “normative reality.” And I find various views in this vicinity pretty plausible, too.

To others, though, such views seem like they’re missing something central. Here I want to examine one thing in particular that it’s not clear they can capture: namely, the sense in which morality has, or appears to have, authority.

I. Paperclips and flourishing

Here’s a toy model of the type of view I have in mind.

Consider the paperclip maximizer: an AI system with the sole goal of maximizing the number of paperclips in the world. Let’s say that this AI system’s psychology is sufficiently rich that it’s appropriate to say that it “cares” about making paperclips, “values” paperclips, that its model of the world represents paper-clips as “to-be-made” or as “calling to it,” etc. And let’s say it would remain a paperclip maximizer even after exhaustive reflective equilibrium, and even upon ideal understanding of all non-normative facts. Call this system “Clippy.”

Unlike Clippy, let’s say, you care about things like flourishing, joy, love, beauty, community, consciousness, understanding, creativity, etc — for yourself, for your loved ones, for all beings, perhaps to different degrees, perhaps impartially, perhaps in complex relationships and subject to complex constraints and conditions. And you would continue to do so, in a more coherent way, after exhaustive reflective equilibrium, and upon ideal understanding of all non-normative facts.

And let’s say, on the view I’m considering, that there isn’t a “value reality,” independent of both you and Clippy, that makes one of you right about what’s really valuable, and one of you wrong. You’ve got your (idealized) values, Clippy has Clippy’s (idealized) values; they’re not the same; and that’s, ultimately, about all there is to it. (Or at least, almost all. I haven’t yet said much about how you and Clippy are disposed to treat agents with different object-level values in various game-theoretic situations — dispositions that may, I think, be crucially important to the issues discussed below.)

Of course, we can dance around some of the edges of this picture, to try to avoid endorsing claims that might seem counterintuitive. If we want, for example, we can try to get more specific about how the semantics of words like “right” and “ought” and “valuable” work in this sort of situation, when uttered by a particular party and evaluated from a particular perspective. We can say, for example, that in your mouth, or evaluated from your perspective, the word “good” rigidly refers to e.g. love, joy, beauty etc, such that it will be true, in your mouth, that “joy is good,” and that “joy would still be good even if I valued clipping,” and/or false, evaluated from your perspective, when Clippy says “clipping is good.” But everything we can say about you can be said, symmetrically, about Clippy. Ultimately, if we haven’t already “taken a side” — if we abstract away from both your perspective and Clippy’s perspective (and, also, from our own contingent values) — there’s nothing about reality itself that pulls us back towards one side or another.

Let’s call views of this broad type “subjectivist.” I’m playing fast and loose here (indeed, I expect that I would class a very wide variety of meta-ethical views, including, for example, various types of “naturalist realism,” as subjectivist in this sense), but hopefully the underlying picture is at least somewhat clear.

To illustrate, suppose that you and Clippy both come across an empty planet. You want to turn it into Utopia, Clippy wants to turn it into paperclips, and neither of you wants to split it down the middle, or in any fraction, or to try to co-exist, trade, compromise, etc (this equal refusal to cooperate is important, I think). So, you fight. From an impartial perspective, on the view I’m considering, this is no different from a fight between Clippy and Staply — a different sort of AI system, that maximizes staples rather than paperclips. You are both righteous, perhaps, in your own eyes. Both of you choose, equally, to try to create the world you want; and because your choices conflict, both of you choose, it seems, to try to impose your will on the other. But the universe isn’t “rooting” for either of you.

II. What’s missing from subjectivism?

Various people, including myself, feel intuitively that this sort of picture is missing something — maybe many things — central to moral life.

One way of trying to cash this out is in terms of things like “disagreement” and “objectivity.” Thus, for example, we might wonder whether various semantic proposals of type I gestured at above will be able to capture our sense of what’s going on when people disagree about what’s good, to be done, right, etc (see, e.g., moral twin earth). Or, relatedly, we might wonder whether this sort of view can capture the sense in which we think of moral norms as objective — e.g., as in some sense independent of any particular perspective.

Here I want to examine a different angle, though — one closely related to disagreement and objectivity, I think, but which has a more directly normative flavor. My question is whether this view can capture our sense of morality as possessing “authority.”

I expect some readers are already feeling themselves un-concerned. Authority, one might think, is some kind of wooly, hard-to-define human construct — the type of thing some people, by temperament, just don’t feel very torn up about having to jettison or substantially revise. And maybe that’s ultimately the right response. But as I discuss below, I also think “authority” closely tied to the notion of “obligation,” which various readers might be more attached to. And regardless, let’s at least look at what we’d be revising, or throwing out.

III. Responding to cruelty

Imagine a group of bored men, who decide, one night, that it would be fun to look for a bit of trouble. One of them mentions a homeless man he finds ugly and annoying, who often sleeps in an alleyway nearby. Wouldn’t it be fun to rough him up? The rest are game. They grab a few beers and head to the alley, where they find the homeless man lying on some cardboard. He’s gaunt, and shivering, and once he sees the group of men approaching him, clearly scared. He gets up, and ask them what they want. They push him down. One nudges him with a boot, and starts taunting him. Others join in, laughing. He scrambles backwards and tries to run, but they push him down again. Someone kicks him in the ribs, and feels something break. He cries out in pain. Another kicks him in the spine. Another picks up a nearby plastic crate, and starts beating him with it.

(I’m describing this in some detail, in the hopes of prompting a fairly visceral reaction — a reaction the flavor of which I’m not sure subjectivism can capture. And of course, in real life, these cruel men, and their victim, would not be thin caricatures of callousness and helplessness, but real humans, with their own lives and complexities.)

Now suppose that I come across these men in this alleyway; that I see what they’re doing, and am in a position to confront them (if it simplifies the case, imagine that I can do so without any risk to myself). I imagine feeling a lot of things in this situation — anger, horror, fear, uncertainty. But I also imagine addressing them, or wanting to, in a very specific way — a way ill-suited both to pursuing my own personal preferences about the world (however idealized), and ill-suited, as well, to giving other people advice about how best to pursue theirs.

In particular, I imagine feeling like I’m in a position to look them in the eye and say something like: “Stop.” That is, to speak with a kind of finality; to stand on a certain type of bedrock. I’m not just saying or signaling: “I don’t want you to do this,” or “he doesn’t want you to do this,” or “I and others are ready to fight/blame/punish you if you keep doing this,” or “you wouldn’t want to do this, if you understood better,” or any combination of these. Indeed, I am not just “informing them” of anything.

But nor, centrally, am I coercing them, or incentivizing them. I may try to stop them, and I may succeed, but I am not merely “imposing” my will on them, as they imposed theirs on the homeless man. My will feels like it is not merely mine, and not even merely mine and his (though his is centrally important). It’s rooted in something deeper, beyond both of us; something that seems, to me at least, ancient, and at the core of things; something in the face of which they should tremble; something whose force should hurl them back against the alleyway walls.

(I think the particular phenomenology I’m describing here may be somewhat contingent. But I think that many common forms indignation, righteous anger, demand for explanation, “what the f*** do you think you’re doing?” and so forth are in a similar ballpark.)

IV. The second person standpoint

What is this “something,” this ground it feels like you’re standing on, in cases like these? I’m going to leave it unspecified for now. But whatever it is, it gives, or seems to give, the homeless man, and those who stand with him against these cruel men, a certain type of “authority.” Not “empirical authority,” in the sense that implies that someone’s volition in a situation — a monarch, a sergeant, a CEO — will in fact be recognized and obeyed. The cruel men may very well not listen; they may laugh in the face of those who oppose them, or react with further violence. Rather, it’s a type of “normative authority.”

What is “normative authority”? Again, I’m not sure. Intuitively, it’s related to a type of external-ness, a kind of “not-up-to-you”-ness, but also to a type of “binding-ness.” The number of socks in your sock drawer isn’t “up to you” either, but that number does not “bind” you in a practical sense. You are “free,” as it were, to pretend the number is otherwise, and to accept the consequences. But you are not, we think, “free” to beat up homeless people, and to accept the consequences. We do not say, to these men, “you can try to beat up this homeless man if you want, but we will try to stop you, and you’d regret the choice if you understood better.” We say: “Stop. This is wrong.”

I’m not saying I have a clear characterization of this. Indeed, as I discuss a bit below, I think characterizing it is surprisingly difficult regardless of your meta-ethics. And what’s more, I think, the notion of an authoritative, binding “must,” or “obligation” can, in some contexts and for some psychologies, be quite harmful, leading to a kind of reluctant and fearful relationship towards an alien, externally-imposed standard, rather than to an effort to understand and take responsibility for one’s actions and their consequences. And of course, there are lots of dangers involved in treating one’s own values and moral beliefs as “authoritative’; perhaps, indeed, humans generally err on the side of too much of such moralizing.

Still, it seems plausible to me that something like this notion of “authority” is core to our basic picture of morality. This is something that Stephen Darwall has written a lot about; he calls the perspective we take up, in addressing these men with statements like “Stop,” the “second-person standpoint” — a standpoint he thinks essentially structured by relationships of authority to address claims (understood as something like moral commands) to one another.

V. Command and counsel

I’m not going to try to lay out or evaluate Darwall’s full picture here, but I do want to emphasize one thing he puts a lot of weight on: namely, the difference between his notion of “second-personal authority” and something like “giving advice” or “providing moral information.”

In attempts to find some sort of rational fault with people acting wrongly, subjectivists often appeal to be various idealization procedures, which would show, it is hoped, that the actions of wrongdoers are mistakes by their own lights. Here I think back to a conversation I had with a subjectivist many years ago, in which she attempted to capture the sense in which what the Nazis did was wrong by appealing to the fact that suitably informed and reflectively equilibrated versions of their values would not have endorsed their actions (on her view, as I recall, what the Nazis really wanted was a certain type of physical and emotional security, which they were pursuing in a very confused way).

One problem with this is that even if true (does our condemnation really depend on this assumption?), it seems very contingent. People sometimes illustrate this by appealing to an “ideally coherent Caligula” (see Street (2009)), who wishes solely and coherently to maximize the suffering of others, and who, it is supposed, would continue to wish this on ideal reflection. To the extent one wishes to condemn Caligula’s actions as wrong, one might think, one needs some standard other than incoherence or factual mistake. (Clippy is another such example, but Clippy may seem too alien for typical judgments of right and wrong to apply.)

More importantly in this context, though, even if we grant that e.g. Nazis, or the cruel men above, are making some mistake by their own lights, addressing them with “Stop” seems very different from informing them of this mistake, or as offering them “moral information.” Darwall, as I recall, talks about this in the context of a distinction drawn by Hobbes between “command” and “counsel.” Thus, you might inform someone that they are stepping on your toe, and that the world would be better, by their lights, if they got off of it — this would be counsel. Or you might tell them to get off your toe. This would be command — command that, in most contexts, you have the authority to make.

What’s more, we might think, someone’s making a practical mistake — even a very high-stakes one — does not necessarily warrant occupying a standpoint of “command” in relation to them. If Bob would really do much better by his own lights by getting that cancer treatment, or saving for retirement, or whatnot, but it looks like he’s choosing not to, this does not, intuitively, give you authority to address him with commands like: “Bob, get the chemo,” or  Bob, open that 401k” — even if you know he’s messing up. Bob is his own person; he gets to make his own choices, however misguided, at least in certain spheres. But beating up homeless people isn’t one of them.

That said, I think the question of when it’s appropriate to occupy some standpoint of practical “authority” in relation to someone’s possible mistakes is quite a complicated one; perhaps it is more appropriate to tell Bob to get the chemo than it might naively seem (especially if you’ll thereby help him a lot). But regardless, it still seems like there’s a conceptual distinction between what you’re doing with Bob, in telling him to get the chemo, and what you’re doing with the cruel men, in telling them to stop.

(I don’t think that using grammatically imperative language is necessary for taking up a standpoint of authority in this sense. For example, I think that saying to these men “what the f*** do you think you are doing?” occupies a similar stance; as do, for example, various forms of unspoken anger. And nor, conversely, does all imperative language indicate such an authoritative stance.)

VI. Is this really about subjectivism?

One issue with subjectivism, one might think, is that it can’t capture an adequate notion of moral authority in this sense. On subjectivism, it might seem, agents can make mistakes in the sense of failing to act in line with their idealized values. And when agents with different idealized values get into conflict, they can trade/cooperate/compromise etc, or they can fight. But it’s not clear what would give any one type of subjectivist agent moral authority in the sense above. Ultimately, it might seem, subjectivist agents are just trying to impose their (idealized) wills on the world, whatever those wills happen to be; in this sense, they are engaged in what can seem, at bottom, like a brute struggle for power — albeit, perhaps, a very game-theoretically-sophisticated one, and one that affords many opportunities (indeed, incentives) towards conventionally nice and mutually-beneficial behavior between agents whose values differ (especially given that agents can value various forms of niceness and inclusiveness intrinsically). Impartially considered, though, no participant in that struggle has any more “authority” than another — or at least, so it might seem. They’re all just out there, trying to cause the stuff they want to happen.

On reflection, though, it’s not actually clear that positing some set of True Objective Values actually solves this problem directly (though it does provide more resources for doing so). To see this, consider a simple case, in which the One True Set of Objective Values is the set posited by totalist, hedonistic utilitarianism. In this case, if you are a totalist hedonistic utilitarian, and Clippy is a paper-clip maximizer, this does, in some sense, privilege you in the eyes of the universe. You are fighting for the “true values,” and Clippy, poor thing, for the false: well done you (though really, you just got lucky; Clippy couldn’t help the utility function it was programmed with). But does this give you the authority to demand that Clippy maximize pleasure? Maybe some versions of Clippy (especially the ones trying to maximize the One True Values de dicto, which isn’t the normal thought experiment) are making a kind of practical mistake in failing to maximize pleasure – but as we saw above, not all practical mistakes give rise to the type of authority in question here. Thus, on this view, one might inform the men beating the homeless person that they are failing to maximize pleasure, which, they should know, is the thing to be maximized — the universe, after all, says so. But this seems comparable to the sense in which, on subjectivism, you might inform them that their idealized values would not endorse their cruel behavior. Perhaps you would be right in both cases: but the sense in which it is appropriate to tell them to stop has not been elucidated.

Of course, this won’t bother the utilitarian, who never put much intrinsic weight on notions of “authority” anyway. But I do think it at least suggests that being “objectivity right” does not, in itself, conceptually entail moral authority in the sense above (though it does seem helpful).

Here’s another way of putting this. On a naive, economics-flavored subjectivism, the basic picture is one of agents with different utility functions and beliefs, and that’s all. The naive realism just discussed adds one more element to this picture: namely, the “True” utility function. Great. But the notion of “authority” — and with it, the notion of “obligation,” or of being “bound” by a standard external to yourself — doesn’t actually yet have an obvious place in either picture. We’ll have an easier time talking about being incorrect or mistaken about the “true values,” if there are such things. But we haven’t yet said why the true values bind — or what “binding” or “authority” or “obligation” even is.

This is related, I think, to the sense in which consequentialists have always had a bit of trouble with the notion of “obligation.” Consequentialists are happiest, in my experience, ranking worlds, and talking about “the good” (e.g., the “True Ranking”), and about which actions are “better.” Their view adds that you are in some sense “required” or “obligated” to maximize the good; but sometimes, it adds this almost as an afterthought. And indeed, if the central normative notion is “betterness,” it’s not yet clear what “obligation” consists in. For example, obligation is plausibly tied, conceptually, to notions of blame, and what is “worthy” of blame. But what sorts of reasons does this “worthiness” encode? Consequentialist reasons for blaming behavior? But these will depend on circumstance, and it won’t always be consequentially-best to blame people for not maximizing the good (as consequentialists widely acknowledge). But the consequentialist isn’t particularly excited about positing non-consequentialist reasons for blame, either, and certainly not about actually giving them non-instrumental weight. So consequentialism, in my experience, tends to incline strongly towards instrumentalist accounts of obligation, centered on the consequentialist pros and cons of different practices of moral blame and praise. But if we go that route, the sense in which the consequentialist thinks maximizing the good obligatory remains quite obscure — despite the fact that this is traditionally treated as definitional to consequentialism. The view seems more at home, I think, in a normative universe that gives the idea of “obligation” little pride of place, and which sticks with “betterness” instead.

But if even very “objective values”-flavored forms of consequentialism don’t provide any ready notion of authority or obligation, this suggests that what’s missing from the picture of subjectivism above isn’t necessarily “objective values” per se, but rather something else. And in that light, it becomes less clear, I think, that subjectivism can’t have this thing, too.

VII. What binds?

I am not, here, going to attempt an analysis of what moral obligation or authority consists in, or what a particularly subjectivist account of it might look like. I’ll note, though, that I’m interested in views on which at least part of what gives rise to the moral authority at stake in cases like the beating described above is something about norms that a very wide variety of suitably cooperative and sophisticated agents have strong reason to commit (perhaps from some idealized, veil-of-ignorance type perspective) to abiding by and enforcing. We might imagine such norms as operative, for example, in some vision of a world in which agents who care about very different things — as different, even, as paperclips and pleasure — manage to avoid the needless destruction of conflict, to act in mutually beneficial ways, and to make adequate space for each to pursue their own ends, conditional on respecting the rights of others to do so as well (see e.g. here). By telling the men to stop, you are, implicitly, including them in this vision, and the community of agents committed to it, and holding them to the expectations this commitment involves. And in this sense, your address to them is a form of respect; they are not yet, in your eyes, beyond the pale, to treated as mere forces of nature, like tigers and avalanches (albeit, cognitively sophisticated ones), rather than members — however norm-violating — of something collective, some sort of “we.”

I’m not sure how far this picture of authority stemming from cooperative norms will really go, but I think it goes some of the way, at least, to ameliorating the sense in which it can seem, on subjectivism, that one is merely imposing one’s contingent will upon the world, or on agents like the cruel men. On this view, in moral contexts like the one above, you are not merely acting on behalf of what you care about, and/or what the homeless man cares about; you are also acting on behalf of norms that agents who care about things in general (or at least, some substantial subset of such agents) have reason to commit to upholding; norms that make it possible, in this strange universe, amidst so much vulnerability and uncertainty, for agents to pursue their values at all. Perhaps the universe itself does not care about those norms; but agents, in general, have reason to care.

I’ll admit, though, that even as I write this, it doesn’t feel like it really captures my intuitive reaction to the cruelty I described above. Really, what it feels like is just: when they kick that homeless man, God roars in anger. What they are doing is wrong, wrong, wrong. It needs to stop, they should be stopped, that’s all, stop them.

Sometimes I imagine learning, against what currently appear to me the strong odds, that some form of non-naturalist moral realism is true. I imagine someone far wiser than I telling me this, and asking: did you really think that it was just some subjective thing, or some sort of complex game-theory decision-theory thing? Didn’t a part of you know, really, the whole time? I imagine thinking, then, about cruelty like this.

But if non-naturalist moral realism is false, I’m not also interested in pretending that it’s true: better to live in the real world. And subjectivism, I think, may actually have resources for capturing certain types of moral authority — though perhaps, ultimately, not all. Regardless, my overall view on meta-ethics, as on many other topics in philosophy, is that our main priority should be making sure human civilization reaches a much wiser state — a state where we can really figure this stuff out, along with so many other things, and act, then, on our understanding.


We should not raise awareness

1 марта, 2021 - 11:46
Published on March 1, 2021 8:46 AM GMT

The phrase "we should raise awareness about x.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} " creeps me out. I had trouble identifying exactly why until I read this summary of simulacra levels.

Level 1: “There’s a lion across the river.” = There’s a lion across the river.

Level 2: “There’s a lion across the river.” = I don’t want to go (or have other people go) across the river.

Level 3: “There’s a lion across the river.” = I’m with the popular kids who are too cool to go across the river.

Level 4: “There’s a lion across the river.” = A firm stance against trans-river expansionism focus grouped well with undecided voters in my constituency.

Level 1 states truth about reality. Level 2 manipulates reality. Level 3 states truth about social reality. Level 4 manipulates social reality.

The transition from Level 1 to Level 2 is trading truth for deception. The transcendence from Level 2 to Level 3 trades physical reality for social reality—an abstraction. The transcendence from Level 3 to Level 4 is trades social truth for social deception.

"We should raise awareness about x" jumps all the way up to level 10.

Everything below this line is my own invention and does not correspond to standard usage of simulacra levels.

Level 5 and Level 6 (Media Creation)

If you want to manipulate people on a grand scale then you must transcend to an even higher abstraction: the media. "The media" is just what we call the target of the masses' attention. Media must be interesting if it is to catch the masses' attention efficiently. Level 5 is about genuine interestingness. Level 6 is about creating the appearance of interestingness.

  • Level 5 (news): "There's a lion across the river." = Lions and rivers are interesting.
  • Level 6 (clickbait): "There's a lion across the river." = Clickbait with the word "lion" in the title maximizes ad revenue for my news outlet.

Levels 5 and 6 are no longer even about getting people on your side (social reality). They are about generating attention for your information delivery system. (The attention can later be commoditized.) Levels 5 and 6 are the realm of reality TV stardom. These levels are about creating channels and brands. The next level is about manipulating channels and brands.

Level 7 and Level 8 (Media Manipulation)

In English, we often use different words for traditional advertising, native advertising, press releases and propaganda. These distinctions matter if you are an ethical reporter. The distinctions are irrelevant to someone who needs to disseminate a message. Since I am an entrepreneur, not a reporter, I use the word "advertising" to mean "calling public attention to one's product, service, need, etc." instead of "paid announcements"[1].

Many people have the idea that news companies send reporters to carefully verify facts. Actually, news outlets usually just republish press releases with a few edits[2]. Even live interviews usually ask predictable softball questions. Interest groups write news and news outlets publish it. Any news outlet which doesn't let advertisers subsidize its stories has trouble competing with competitors who do.

  • Level 7 (press releases): "There's a lion across the river." = This press release was written by a company selling boats and big game rifles.
  • Level 8 (propaganda): "There's a lion across the river." = The Zebra Party owns your communication infrastructure.
Level 9 and Level 10 (Pure Memetics)

Memes are often created by people, usually people with specific interests. Once released, memes self-replicating. They mutate and are selected. They evolve.

We started with facts. Then moved up to alliances. Then media. Then propaganda. As memes evolve, they separate from human interests. No longer does the meme manifest a human being's intention. The meme is trying (in the Darwinian sense of the word "trying") to survive and replicate.

  • Level 9: "There's a lion across the river." = The sentence "There's a lion across the river." has high memetic fitness.
  • Level 10: "We should raise awareness of lions on the other side of the river." = The "There's a lion across the river." meme is trying to mind control you so it can replicate.
  1. Both of these definitions come from Dictionary.com. ↩︎

  2. My understanding of media manipulation comes from historical precedent combined with my firsthand experience. For example, I started a company with a press release so good the editor of TechCrunch used it as a case study in how to write press releases. ↩︎


Pacific Time Zone Discussion

1 марта, 2021 - 10:35
Published on March 1, 2021 7:35 AM GMT

This is a online community get-together scheduled at a convenient time for Less Wrong users in the Pacific Time Zone. You're welcome to join whether or not you live in the Pacific Time Zone.

There is no official topic for this event. There is no required reading. There are a few things I've been thinking about and would like to discuss. You are welcome to add your own to the list.

We'll use Google Meet https://meet.google.com/quw-ioks-tpc


Introduction to Reinforcement Learning

1 марта, 2021 - 07:23
Published on February 28, 2021 11:03 PM GMT

(cross-post from my blog)

This post is the first in a series of notes on Reinforcement Learning.

In the past few months, I have been learning about this topic, and I have found it completely fascinating. Aside from the computational beauty and usefulness of the algorithms, I find that it provides an interesting framework to think about how I approach learning from experience in my own life.

Reinforcement Learning is a branch of AI that tries to develop agents that can learn interactively from real experience. The intelligent agent is assumed to have a set of sensors by which it can observe the world, a set of actuators by which it may act on the world, and a way to perceive a reward signal. The reward signal may be positive or negative, telling the agent whether its behavior is desirable or undesirable.

The reinforcement learning problem is fundamentally different from supervised learning problems such as computer vision. In computer vision, the goal is to learn a mapping from images to labels, and there is a clear error signal at every attempt: the difference between the proposed label and the true label. This means that computer vision has a clear specification of how to do the task. In contrast, reinforcement learning begins with a high-level description of “what” to do, but allows the agent to experiment and learn from experience the best “how”. This is especially useful in situations in which we humans may not be able to clearly articulate how we perform a task, such as playing chess, which relies heavily on intuition at each move, cases in which we expect the AI system to be able to eventually outperform human knowledge, or even cases in which we do not know how to do the task at all, such as folding complex proteins.

The key challenge is that rewards are usually given as a function of being in a specific state, but the agent may need to exhibit complex behaviors over time to be able to achieve such a state. For example,

  • A chess-playing agent will receive a reward of +1 for checkmating its opponent, -1 for being checkmated, and zero for any other circumstance. However, the agent must actually play an entire game of chess before it may observe any actual rewards. Somehow, it needs to learn which types of moves in which positions are more likely to lead to victory.
  • A cleaning robot receives a reward every time it completely cleans a room. However, the robot needs to completely sweep and mop the floors before it can see any reward. Somehow, the robot must learn sequences of behaviors that perform cleaning tasks.

This leads to several challenges that are unique to the field of reinforcement learning:

  1. Exploration. The learning agent needs to find out how to obtain rewards. It literally needs to try to do random things until it starts to see rewards. Then, it can start to optimize for those rewards.
  2. The exploration-exploitation trade-off. Once the agent has started to see rewards, we are immediately faced with a dilemma: how aggressively should the agent optimize for the rewards it has seen, versus how much should it continue to explore to search for potentially greater rewards?
  3. The credit assignment problem. As the agent interacts with its environment and receives rewards, how can it determine which aspects of its behavior were most relevant to the observed rewards? This is the problem of correctly learning which sequences of states and actions lead to good outcomes
  4. Reward hacking. Reward hacking is Goodhart’s Law applied to AI. Sometimes, when we create a reward function for our learning agent, we find that we did not clearly articulate the exact thing we wanted the agent to learn. In general, correctly specifying the behavior we want is a surprisingly difficult problem. For example, if we have a cleaning robot, we may reward it for every piece of trash it puts into a trash can. Reward hacking will result in a robot that creates its own trash by destroying objects in the room, then putting them into the trash can to collect its reward.

All of the challenges described above are central to both the algorithmic complexity of RL, as well as to the difficulties of developing safe learning agents. When a cleaning robot is exploring the actions it can take in a new room, we would like to make sure that it will not try to stick its body parts into an electric outlet, which could result in a destructive fire. An agent that fails to properly exploration and exploitation could result in either a robot that is “depressed”, sitting in a corner gathering small amounts of reward, or a robot that is “restless”, always exploring and never committing to anything. Incorrect credit assignment can lead to behaviors that are remarkably similar to superstitions. Reward hacking can famously result in paperclip manufacturing engines that dismantle the solar system in their quest for making more paperclips.

The next post in this series will talk about the assumptions and framing of reinforcement learning. We’ll discuss intuitions around the Markov property, and the decomposition of the problem into policies and value functions.


Takeaways from one year of lockdown

1 марта, 2021 - 06:53
Published on March 1, 2021 3:53 AM GMT

As of today, I've been in full-on, hardcore lockdown for an entire year. I have a lot of feelings – both about the personal social impacts of lockdown and about society being broken – that I won't go into in this public space. What I want to figure out in this post is what rationality-relevant lessons I can draw from what happened in my life this past year. 

(Meta: This post is not well-written and is mostly bullet points, because the first few versions I wrote were unusable but I still wanted to publish it today.)


Some facts about my lockdown:

  • I have spent 99.9% of the year within 1 mile of my house
  • Up until last month I had spent the entire year within 10 miles of my house
  • Between February 29th and June 15th of 2020, I did not set foot outside of my front gate
  • I have not gotten COVID, nor has anyone in my bubble
  • I have incurred an average of 0 microCOVIDs per week
  • The absolute riskiest thing I've done this whole time cost ~20 microCOVIDs
  • I can only remember talking to a friend not-in-my-bubble, in person, twice

Some observations about other people with similar levels of caution:

  • Almost no one I know has caught COVID, even though Zvi estimates that ~25% of Americans have had it (the official confirmed rate is 10%). I know of only one person who caught it while taking serious precautions, and I know a few hundred people about as well as I know this person. (see also)
  • I was recently tracking down a reference in the Sequences and found that the author was so afraid of COVID that he failed to seek medical care for appendicitis and died of sepsis.

On negotiations:

  • A blanket heuristic of "absolutely no interactions outside of the household" makes decisions simple but is very costly in other ways
  • microCOVID spreadsheets are useful but fairly high-effort
  • I went on a date once. The COVID negotiations with my house were so stressful that I had a migraine for a week afterwards.

On hopelessness:

  • I spent a fair amount of time trying to get vaccinated early, and failed. I now appear to have a belief that I will never succeed at getting vaccinated; and further that other people can succeed but I never can. 
  • Related: My system 1 believes that lockdown will last forever. Also that vaccines aren't real – not that they don't work, but that they're a lovely dream, like unicorns or God, that ultimately turns out to be a lie. A vaccine cannot cause me to leave lockdown because lockdown is an eternal, all-consuming metaphysical state.
  • I would have liked to be dating this year, but the first date and the surrounding ~week of house discussion was so stressful that I gave up on dates entirely after that.
  • I notice that I feel the lack of friendships, but wasn't motivated enough about any particular friendship to put in the effort to make it work despite the situation. By contrast, some people I know did do this and have benefited a lot.
  • My house had ~3-hour meetings ~3 times a week at the very beginning of the pandemic, where people did math on the board and talked about their feelings and we tried to figure out what to do. In retrospect, this burned me out so much that I gave up on trying to figure anything out and defaulted to an absolutely-zero-risk strategy, because at least that was simple.
  • The fact that SlateStarCodex went down at the same time everything else in life went to shit destroyed my soul.
  • Oops I have started talking about feelings and will now stop.

Taking all these observations together, it's clear to me that my social group has been insanely overcautious, to our great detriment. I think this has been obvious for quite a while, but I didn't and still don't know how to act on that information.

It seems like extreme caution made sense at first, when we didn't know much. And by the time we knew the important, action-relevant information like transmission vectors and all that jazz, we were already used to being afraid, and we failed to adjust. Looking back at case counts, my behavior in the summer was completely unreasonable – I felt afraid of my housemates going on walks while wearing masks!

So one blocker on expanding the range of actions we were taking was that we'd gotten used to it. Another blocker was that, even if I were to have gone back to living my life as normally as I could, that would still not be very normal. I think it didn't seem that worth it to me to take any risks at all, as long as I couldn't have my life back anyway. I briefly entertained the idea of a Berkeley rationalist 'megabubble', but backed off when case counts went up again, someone I knew got long COVID just from outdoor, masked, distanced socializing, and also I realized that it would just be really a whole fucking lot of work to coordinate.


Here are some takeaways re: rationality:

  • Trivial inconveniences are enough to make it so that you never do anything at all
  • I underestimated how hard it is to get your system 1 to understand numbers
  • I underestimated how quickly my new normal would ossify / how difficult it would be to modify my behavior in response to new information
  • I think there's a reasonable level of COVID caution that is not just "live life as normal", but rationalists have been COVID-cautious to an extent that far surpasses that and wraps around to being harmful again (with the sepsis death as an extreme example)
    • This is especially true because most rationalists are in their 20s and 30s, and so, barring other risk factors, it would probably have just been fine for most of us to get COVID
    • You might say "but tail risks!", which, yes, is most of the reason for trying to avoid getting COVID (the other reason being spreading). My response to that is "but a whole year of destroying huge amounts of value!" It's not clear to me which way the scales would tip here; I haven't tried to estimate it.
  • It's way harder to be a good rationalist in stressful situations. This is a point I've seen discussed elsewhere on LessWrong but I can't remember where.
  • Not tied to the above observations, but – I think that despite rationalists being ahead of the curve on taking precautions, figuring out the nature of the virus, etc., we developed a kind of learned helplessness around being in lockdown; we, like everyone else, have just been waiting for it to end. 
    • I think if we had been more serious about taking matters into our own hands, more of us would have gotten interested in RaDVaC sooner (even if you don't think it works, we still should have discussed it earlier), and more of us would have signed up for vaccine trials.
    • The creation of microcovid.org is a counterexample, since it materially helped many people I know to return to some semblance of a normal life.
    • Generally I think we frontloaded our effort and then ran out of steam. Not clear that it was wrong to go really really hard at the beginning, but it ended up being costly in the long run.
  • Even people as critical of official recommendations as we are can still be swayed by them. My example is that due to the surgical and cloth mask propaganda, it didn't even occur to me to wear the P100 I already owned until many months into the pandemic. Noticing that I could do this would have greatly expanded the range of actions I felt comfortable taking.
  • Negotiating in emotionally fraught situations is a very difficult skill, and despite all the training they receive in talking about feelings and what-not, being a CFAR instructor does not make you good at this skill (source: almost everyone in my house was a CFAR instructor or mentor).
  • Sometimes I just didn't think of things, and I'm not sure what would have helped with that (maybe fixing my GTD system sooner?). I didn't send my parents vitamin D until September despite taking it myself the whole time, and I didn't realize my mom didn't have a P100 (and had only been wearing surgical masks to grocery shop!) until December(!).
    • Maybe I would have caught these if I had just set a five-minute timer and written down all the things I should be doing to protect my parents. I did not do this or even think about doing it.
    • A more generalizable takeaway is perhaps that it's easier to remember to act when there's an acute problem than when there's a chronic one. Perhaps pay more attention to the chronic issues in your life?
  • I underestimated how powerful fear is (as a motivator, as a force in negotiations, etc)

I have now run out of time to write this post, which is probably for the best since it is just rambling at this point. I will end with a quote:

Sometimes you go through things that seem huge at the time, like a mysterious glowing cloud devouring your entire community. While they’re happening, they feel like the only thing that matters, and you can hardly imagine that there’s a world out there that might have anything else going on.

And then the Glow Cloud moves on. And you move on. And the event is behind you. And you may find, as time passes, that you remember it less and less. … And you are left with nothing but a powerful wonder at the fleeting nature of even the most important things in life, and the faint but pretty smell of vanilla.


When I left Google

1 марта, 2021 - 00:40
Published on February 28, 2021 9:40 PM GMT

About five years ago a friend had recently started at Wave, and encouraged me to join them. It sounded like a great startup doing really important work in East Africa, and potentially more valuable than continuing my "earning to give" approach, but because it was a fully remote company and I like being in the same place as my coworkers I told him I wasn't interested.

(With the pandemic, after nearly a year of working remotely for an "in person" company this is still a strong preference!)

About a year later we ended up talking again, and there were now several other Wave engineers in Boston; if I joined I would potentially be the fourth engineer here. We could get a coworking space, possibly hire more local people, and I wouldn't need to be entirely remote.

I was still unsure about the remote aspect, and whether four Boston engineers would feel good, but I already knew the three others socially, and it seemed promising. We spent some time together and I decided to interview. Throughout the interview I was thinking about fit: would I like working at Wave? Would I work well with people? Did their current engineering staff and I have similar enough ways of thinking about it?

They ended up deciding to make me an offer, and I accepted! I planned to stay at Google until I got my annual bonus, and then switch.

A few weeks later, though, I realized something and my stomach dropped: I hadn't properly thought through this decision from an impact perspective. Yes, it seemed plausibly very important, but how sure was I? Especially compared to GiveWell's recommendations which had stood up to a lot of scrutiny? I had made the mistake of letting "it's remote" expand subconsciously from being a blocker to being my only reason for not taking the job.

I felt very stuck. If it wasn't actually better than what I was already doing I really shouldn't switch. I'd already accepted the job, and I don't like going back on my word, but I still could if I really needed to.

As it was I was in luck. Days earlier, Tavneet Suri and William Jack had published "The long-run poverty and gender impacts of mobile money" (pdf) which looked at almost exactly the question I was interested in. I dug into it, wrote a draft of Estimating the Value of Mobile Money, and ran it by some friends for help. After working through the numbers it looked like the work I would be doing at Wave was, if we succeeded, really valuable stuff. I did go ahead with the transition, though it ended up not working out. [1]

I'm still open to potentially moving away from earning to give at some point (especially if the money I would be donating is going to be lost to means testing), but next time I'm considering making such a big decision I need to be much more careful to thoroughly think it through first.

[1] My options did end up being worth something, unlike the previous time, though I don't fully know how much yet.

Comment via: facebook


The Puce Tribe

1 марта, 2021 - 00:11
Published on February 28, 2021 9:11 PM GMT


Potentiated by: https://www.lesswrong.com/posts/GLMFmFvXGyAcG25ni/i-can-tolerate-anything-except-the-outgroup
Inspired by: https://www.lesswrong.com/posts/LundKP6HuuC9CiaWw/naming-the-nameless

Finally triggered by: https://astralcodexten.substack.com/p/book-review-fussell-on-class and https://alexdanco.com/2021/01/22/the-michael-scott-theory-of-social-class/ 

And in case you need a refresher:

The Red Tribe is most classically typified by conservative political beliefs, strong evangelical religious beliefs, creationism, opposing gay marriage, owning guns, eating steak, drinking Coca-Cola, driving SUVs, watching lots of TV, enjoying American football, getting conspicuously upset about terrorists and commies, marrying early, divorcing early, shouting “USA IS NUMBER ONE!!!”, and listening to country music.

The Blue Tribe is most classically typified by liberal political beliefs, vague agnosticism, supporting gay rights, thinking guns are barbaric, eating arugula, drinking fancy bottled water, driving Priuses, reading lots of books, being highly educated, mocking American football, feeling vaguely like they should like soccer but never really being able to get into it, getting conspicuously upset about sexists and bigots, marrying later, constantly pointing out how much more civilized European countries are than America, and listening to “everything except country”.

(There is a partly-formed attempt to spin off a Grey Tribe typified by libertarian political beliefs, Dawkins-style atheism, vague annoyance that the question of gay rights even comes up, eating paleo, drinking Soylent, calling in rides on Uber, reading lots of blogs, calling American football “sportsball”, getting conspicuously upset about the War on Drugs and the NSA, and listening to filk – but for our current purposes this is a distraction and they can safely be considered part of the Blue Tribe most of the time)

The label “Gray Tribe” is from the ancient days of 2014 and it feels outdated to me now.

I’m gonna write about a fictional Puce Tribe. Why puce? Because it sounds funny, that's why. The following attributes are meant partly as aspiration and partly as realistic speculation.

  • Unlike the gray tribe, the puce tribe cannot be “safely considered part of the blue tribe most of the time”. They may or may not be equidistant from blue and red, but they are definitely their own distinct cluster.
  • People of the puce tribe are especially likely to know how to
    • use a fire extinguisher
    • constructively express their feelings
    • field-dress a deer[1]
    • write code
    • speak multiple languages
    • understand subtle statistical arguments
    • troubleshoot power tools
    • overcome or at least manage device addiction
  • The puce tribe is politically diverse, not riven by quite the same political tugs-o-war du jour as everyone else. They are in some ways similar to the gray tribe, but have a more systems-and-incentives-level approach to things. They tend to say things that sound like “blah blah complex problem blah blah bad equilibrium blah blah prediction markets, incentive engineering, small-scale experimentation.”[2]
  • The Puce people know that toughness is a virtue, and they live as such. They do not train weakness into themselves the way those other tribes do. Red tribers think they are tough, but they make themselves soft with physical inactivity and overindulgent diets. The blue tribe systematically rewards complaining and thin skin, to the detriment of their emotional resilience. Meanwhile, gray tribespeople cloister themselves in climate-controlled bubbles and get upset when their lunch delivery doesn’t come with the exact right amount of their preferred condiment.
  • Puce tribespeople feel embarrassingly overindulgent when their ratio of {time spent writing stuff} to {time spent consuming news and ephemera} falls too low.
  • Puce tribespersons are known to come home dirty, marked up, and bruised from a game of paintball.
  • When you walk into a puce home, you have an especially high chance of seeing a stockpile of food, fuel, and emergency supplies tailored to the locals risks (earthquakes, extreme weather, blackouts, fragile supply lines, etc.). Generators and handcrafted solar stills confer especially high status.
  • If you walk through a neighborhood that is both Puce-heavy and suburban, you are likely to see backyards with trampolines, treehouses, or pools--none of them neglected. In garages, you can expect to glimpse soldering stations, 3D printers, DIY projects, sports gear, camping gear, and workshops. Rose gardens not so much.
  • Pucians respect the ritual magic of going out for a beer with the fellas, however dull and low-class that might sound to the blue and gray tribes. The puce tribe also has great reverence for psychedelics, unlike the red tribe who have nothing but self-righteous scorn for all that depraved hippy shit.
  • Physical activity is highly valued and seen as essential to the puce lifestyle, unlike the often sedentary gray tribe. Particular types of exercise vary widely depending on particular tastes and local amenities. Hiking is almost the default activity for those who live near a trailhead. Parkour and soccer are popular in places with mild climates. Dodgeball and badminton are popular where courts are available. Even the humble walk-and-talk is a venerated pastime. Puce friends bump into each other at the gym as often or more than at the grocery store. They are body-positive and have norms of positive reinforcement around fitness.
  • There is no distinctively Pucillanimous diet. The only common feature of the way they eat is purposefulness. Diet is part of the bedrock of a good life, and these people damn well treat it that way. The puce vegetarians explore the multi-armed bandit of veg restaurants, always ready for disappointment and always strong enough to keep looking. The paleo-eaters and keto-eaters meticulously plan their meals for the week, accepting this cost as the Tribute charged by Mother Nature. The cheeseburger lovers don't eat cheeseburgers every day, they space them out among healthier and more fibrous meals (enhanced of course by post-exercise endorphins). The Soylent freaks are happy to save time and cognition for things they care about more. Snacks are often seen as low-status unless they are baby carrots. Diet fads are appreciated for what they are--voluntary experiments that probably won’t work out for you, but can still be worth a try. The puce are conspicuously skeptical of diet science, and this has become something of a self-indulgent shibboleth for them.
    • But also pucesters go nuts for donuts.
  • Young pucemen grow up knowing the energetic, carefree sublimity of a well-timed chestbump. (Sometimes neither a hug nor a high-five is quite enough, and that's why we have chestbumps.)
  • The puce tribe recognizes that the blue tribe is onto something with athleisure and the red tribe is onto something with workwear. Both are popular among the puce, and their overall fashion is not particularly distinctive.
  • The Puce Tribe likes...libraries?? Yeah, that’s right. They consider device addiction to be a particularly vicious vice and they adapt accordingly. It is not uncommon for Puce to bike 15 minutes over to the local library with no phone or laptop, and spend an hour or two giving their full attention to a book they want to understand.
  • NASCAR? Art exhibits? Nah dude. Bounce houses and escape rooms.
  • The Folsom Foundry in SF has “Game Nights” (or at least used to). You show up, pay a fee to get in, and then you can buy pub food or expensive craft beer. But that’s not the real allure. The reason you’re there is because the walls are lined with video game consoles, the middle of the floor is dominated by beer pong tables, and the corner has a group of benches around a projector screen and a Rock Band setup. The lights are low, the music is loud and surprisingly good, and the whole venue is filled with people (somehow at a half-decent gender ratio). Pucefolk don’t waste their money on exclusive restaurants with fancy napkins, nor on addictions to McDonalds. They use their discretionary dollars to demand more things like the Folsom Foundry.
  • At about the same time as the cultural collapse of Silicon Valley in 2022-2023, a new medium-sized tech hub sprang up in Plagueis, Kansas, which already had a thriving puce culture. Apps and webpages from the cool new Plagueis Plains Special Economic Zone have a noticeably different æsthetic from those of the Valley. They still like minimalism, but it’s not flashy minimalism--color schemes are not garish, and you’re never made to scroll unnecessarily. APIs and advanced control panels abound. On the Plain it is considered obviously sleazy to enforce an addictive/manipulative UX. Plaguis has a major crypto streak, and its UX æsthetics have come to be associated with more user-empowerment, often at the cost of prettiness and interpretability. (See again: Naming the Nameless)
  • The Puce are a cultured people--deliberately so. They heed Bryan Caplan’s advice: prioritize entertainment that a few people go crazy for above entertainment that the masses moderately enjoy. In a puce tribe gathering, it would not be at all strange to hear a spirited discussion of the environmental storytelling mechanics of Portal vs Journey, or speculation on what factors might have underpinned the sitcom renaissance of 2009.[3]
  • It is hard to pin down what kinds of music prevail among the puce tribe--tastes are simply too diverse these days. One weird thing does stick out though. The Puce are suckers for those tasty, Westernized erhu tracks (or other asian-sounding string instrument). They like it in gentle backup melodies, brief transitional accents in DnB, or in spunky shōnen intro-type songs. We don't know why. The anthropologists are stumped.



[1] https://www.dickssportinggoods.com/p/hunters-specialties-butt-out-2-16hspabtttxxxxxxxhbg/16hspabtttxxxxxxxhbg

[2] See: Complex Problem, Bad Equilibrium, Prediction Markets Incentive Engineering, Small Scale Experimentation. Also Credible Neutrality.

[3] See: Parks & Recreation, Community, Archer


How do you run a fit-test for a mask at home when you don't have fancy equipment?

28 февраля, 2021 - 23:30
Published on February 28, 2021 8:30 PM GMT

Commercial fit-testing for masks needs quite fancy equipment like the Allegro Saccharin Fit Test Kit 2040 which combines undesireable features of being currently unavailable to order, expensive and having to take up room in my apartment after beign ordered. 

It seems to me that there should be a way to get fit-testing done without as fancy equipment as it's just about exposing oneself to the smell of one of the substances that are are suitable for fit-testing (that have the right molecular size). Has anybody here found a way to do reliable fit-testing for themselves without the commerical equipment?


02/28/2021 - Myanmar Diaries; Context

28 февраля, 2021 - 23:00
Published on February 28, 2021 8:00 PM GMT

Content Warning: violence, depressing content, politics

This is the start of a weekly diary on the Myanmar coup and revolution, inspired by Zvi's weekly Covid blog and Robin Hanson. Every week I will discuss recent events in the conflict between the military regime and the pro-democracy protests. I will make predictions about a few major events, such as state-protestor violence, information control, and the long run distribution of political power. I will focus on simple game theoretic models that explain the internal dynamics of rule. I am a PhD student in comparative politics at Georgetown University, where I specialize in authoritarianism. I hope to devote 4 hours a week to this project.

In this installment I will provide a brief historical context.

Major Context in Myanmar

Three historical periods are critical to understanding the current conflict. The military rule period lasted from 1961 until ~2009. The period began with a coup by general Ne Win overthrowing the parliament and establishing a nominally socialist regime. In the late 2000's the military reformed for internal reasons, establishing power sharing with the opposition lead by Aung San Su Kyi. The current period began in 2021 when the military reasserted control over the state and imprisoned the opposition, fearing they were losing power.

Military Rule 1961-2008

Military defacto power: The entire state apparatus, all extractive industries.

Opposition defacto power: None

Pleased groups: The military, clients of the military

Displeased groups: The rural poor, the middle class, business elites, the opposition, organized religion, the urban poor, the separatist rebels

Rather than provide a full account of the next 50 years of political economy, here are 3 stylized facts about the period of unitary military rule.

  • Expropriation: After seizing power, the military aka the Tatmadaw, expropriated the most lucrative private businesses. The Tatmadaw in particular targeted lucrative mineral extraction industries, including jade, gold, amber, natural gas, and oil, forming Myanmar Economic Holdings Limited (MEHL) to manage these lucrative rent opportunities in 1994. Critically, natural resources freed the state from dependence on taxation. Without an interest in GDP the Tatmadaw regime had little incentive to grow the real economy or provide public goods in exchange for taxes, resulting in a weak state-society relationship. Those natural resources also incentivized particularist rebels to fight for control in the mineral-rich periphery, the origin of the Rohingya refugee crisis.
  • Weak elite coalitions: Most autocracies rely on an alliance between violence specialists, traditional leaders, a section of the middle class, a competent bureaucracy and a business elite. Each group contributes critical resources to the regime, from ideological legitimacy to tax money to information on societies needs. For example, the Saudi regime has long benefited from a tight alliance between a hereditary military elite (the mujahideen), state supported clerics, and powerful business families of Saudi and Palestinian origin. A regime may not possess each component, but the strongest do. From the beginning, the Tatmadaw regime was exceptionally narrow and could only rely on support from violence specialists. The Tatmadaw could pay or coerce other social actors for support, but such support depended on the militaries limited ability to fund and verify reciprocity.
  • Repression and limited collective action capacity: Not surprisingly, state society relations often turned violent during military rule. Public protests for reform were repeatedly met with massacres. At the same time, the state systematically disrupted the people's ability to organize in response. Until the mid 2000's sim cards were priced at hundreds of dollars to prevent citizens from communicating freely. In 2008 a typhoon struck Myanmar causing mass starvation and displacement. The Tatmadaw responded by delaying or stopping aid shipments, refusing any aid workers access to the country, and even dispersing refugees and breaking up internal refugee camps. Foreign observers often remark that the regime was blind to the victims suffering, but this is incorrect. The regime was well aware of the price of its action, but chose to prevent refugee camps and mutual aid form happening. Refugee camps and relief societies are viable cites of collective action in which toe people could organize to overthrow the regime (as occured in Mexico in the 1980's), and the regime chose to disperse the peasants rather than risk it. ~140,000 people died.
The power-sharing period 2008-2021

Military defacto power: Foreign policy, internal security, military policy, all extractive indistries, veto over legislation, patronage networks

NLD defacto power: Domestic policy (education, infrastructure, etc.), veto over legislation, electoral dominance

Pleased groups: Roughly everyone else, separatist rebels

Displeased groups: Eventually the military and their clients became displeased

In 2008 the Tatmadaw began devolving power to the opposition National League for Democracy, lead by Aung San Su Kyi. It appears the military hoped to escape international sanctions and demoralizing internal conflict by giving a minor, controlled role for the opposition. Meanwhile, they would retain their lucrative mineral holdings, a monopoly on violence and the ability to coup the NLD at any time and return to power. The resulting constitution is mostly a "fake", with a parliament hamstrung by constitutional restrictions and seats reserved to the military. Political scientists refer to such states as "hybrid regimes" because meaningful competition is allowed, but the state narrowly contains competition through media control, red-line issues, patronage and continued repression.

Life in Myanmar radically changed under NLD rule. The NLD passed a raft of badly needed economic reforms, opening the country to investment, trade and development. From 2009 to 2010 foreign investment increased from USD 300 million to 20 billion. International aid agencies returned for projects and billions in national debt were forgiven. Standards of living increased drastically: Real purchasing power doubled over ten years from 2009 to 2019. By 2018, 98% of the population owned smartphones and 10% enjoyed a bank account. It is difficult for a western audience to imagine standards of living doubling and modernity arriving in just 10 years.

Unsurprisingly, the NLD became much more popular than the Tatmadaw. To compete, the Tatmadaw formed its own party the Union Solidarity and Development Party (USDP). despite a well-funded patronage systme, the USDP could not compete with the NLD who won repeated landslide elections in 2012, 2015 and 2021. Meanwhile the NLD alienated its western backers by supporting ethnic cleansing of the Rohingya people, which was internally popular. They also ingratiated themselves with the Chinese, Myanmar's main foreign backer.

The 2021 Coup

The Tatmadaw saw itself as increasingly vulnerable beside the popular and united NLD. The NLD reduced compromising with the Tatmadaw after winning control over both houses of parliament in 2015. Eventually the Tatmadaw feared they would lose their ability to launch a successful coup as the NLD conslidated its power and legitimacy.

We can model this interaction as a simple two stage game, see below. The Tatmadaw chooses first to launch a coup and repress the NLD, or continue sharing power. If they repress they can keep the mines, but lose the benefits of power sharing. This is T3, where the military gets 5 utils and the opposition 0 (because they are imprisoned). The military would prefer that the NLD continue running the country but let them keep the gold and other mines. This would maximize their financial power, military power and avoid the status loss from massacring monks. This outcome is represented by T2 where the military gets 10 utils and the NLD gets 5 utils. The NLD prefers continued power sharing because they are not in prison and have power, but they would prefer to control the mines. They also dislike that the same people who murdered their friends get to keep a their stolen gold, jade and oil.

The second stage occurs once the Tatmadaw has lost its coup-launching power (due to NLD popularity, a new generation of officers or the aging of current loyalists). At this stage the NLD chooses between enacting democracy and taking away the militaries stolen mines or continuing to share power. In T1 the NLD gets 10 whole utils because they control the natural resources, the state policy, and a more just outcome. They can choose to continue power sharing, which the military wants, but have no reason to. All the money, power and justice is under the democracy decision.

Therefore, the NLD cannot credibly commit to sharing power. They can say they will share power (cheap talk) but the Tatmadaw will not believe them. As a result, the only Nash equilibrium is repression, which the military chose on Feb. 1st (one day before the next parliamentary session began).

On that day, the military stormed the parliament building in a convoy of APCs. They imprisoned all current NLD leaders including Aung San Su Kyi and the president. They shutdown all opposition media, declared martial law, and broadcast 24hour military propaganda. They claim that the recent election was fraudulent and they will hold a new election in one year. Unfortunately, this promise is also not credible.


Next time we will review the protests so far. For now, here are some contextless predictions.

Min Aung Hlaing is Myanmar head of state on January 1st, 2023 - 55%

The NLD is head of state on January 1st, 2023 - 35%

Pro-Tatmadaw actors kill >20 protestors 02/28-03/06 - 70%

End of protests 02/28-03/06 - 15% (defined as no protests gathering 1000 or more participants in the week)


How might cryptocurrencies affect AGI timelines?

28 февраля, 2021 - 22:16
Published on February 28, 2021 7:16 PM GMT

Summary: Deflation (or so I heard) is considered harmful because it stifles growth. Central banks have been fighting it to keep economies healthy. Cryptocurrencies can be designed to be deflationary. If the market capitalization of cryptocurrencies becomes big, central banks may have no way to contain the deflation. This may be catastrophic in many ways – but might it also slow AGI development and buy safety research more time?

I’ve been wondering what the “crypto endgame” may look like. Crypto may just turn out to have been a bubble or may continue for decades unchanged at its current limited level of significance. But we’re probably sufficiently prepared for those scenarios already.

Instead the crypto market capitalization might take off at a superlinear pace. Bitcoin is currently on rank 14 among the world’s currencies and holds 0.7% of the aggregate value of all of them. All cryptocurrencies together hold about 1.1% of the world’s monetary value.

The crypto space is rife with network effects. Or it is one huge network effect. These are wont to show superlinear growth if they show any growth. Superlinear growth often doesn’t last long enough to be significant, but in other cases it does. So maybe crypto’s market capitalization will grow by an order of magnitude or more within a few years, so before AGI is developed. That might happen if more more companies follow the examples of MicroStrategy or Tesla and invest their spare cash into a cryptocurrency. Phil Bonello, director of research at Grayscale, also mentions a scenario in which governments start buying crypto and thereby force all other governments to follow suit. I don’t know the reasoning behind that, so I don’t know how forcible that effect will be, but it doesn’t seem to be hard to come up with plausible growth scenarios. (At least if you’re incentivized to come up with them the way Grayscale is.)

Some cryptocurrencies, notably Bitcoin, are designed to be deflationary. If companies across the board find that they generate greater profits by investing all their spare cash into deflationary cryptocurrencies rather than reinvesting into their own production, innovation may stall. (Isn’t that one of the typical reasons why deflation is considered harmful?)

So that brings me to my questions:

  1. Conditional on the cryptocurrency market growing by another 1–2 orders of magnitude in the next five years or so, what happens to the global financial system and the world economy? (Might this future look like a cyberpunk dystopia where a rich elite retires on their capital gains, and anyone who didn’t enter the period with enough capital has very few avenues to sell their labor and so lives in poverty?)
  2. And might this stalled innovation also carry over to AGI in that technology companies will find it more profitable to invest into cryptocurrencies rather than compute or staff, thereby buying safety research a few more years or decades to solve ethics, decision theory, and international coordination?


"If You're Not a Holy Madman, You're Not Trying"

28 февраля, 2021 - 21:56
Published on February 28, 2021 6:56 PM GMT

I've been reading the hardcover SSC collection in the mornings, as a way of avoiding getting caught up in internet distractions first thing when I get up. I'd read many of Scott Alexander's posts before, but nowhere near everything posted; and I hadn't before made any attempt to dive the archives to "catch up" to the seeming majority of rationalists who have read everything Scott Alexander has ever written.

(The hardcover SSC collection is nowhere near everything on SSC, not to mention Scott's earlier squid314 blog on livejournal. I'm curious how much shelf space a more complete anthology would occupy.)

Anyway, this has gotten me thinking about the character of Scott Alexander's writing. I once remarked (at a LessWrong meetup) that Scott Alexander "could never be a cult leader". I indented this as a sort of criticism. Scott Alexander doesn't write with conviction in the same way some other prominent rationalist authors do. He usually has the attitude of a bemused bystander who is merely curious about a bunch of things. Some others in the group agreed with me, but took it as praise: compared to some other rationalist authors, Scott Alexander isn't an ideologue.

(now I fear 90% of the comments are going to be some variation of "cults are bad")

What I didn't realize was how obsessed Scott Alexander himself is with this distinction. Many of his posts grapple with variations on question of just how seriously we can take our ideas without going insane, contrasting the holy madman in the desert (who takes ideas 100% seriously) with the detached academic (who takes an intellectual interest in philosophy without applying it to life).

  • Beware Isolated Demands for Rigor is the post which introduces and seriously fleshes out this distinction. Scott says the holy madman and the detached academic are two valid extremes, because both of them are consistent in how they call for principles to be applied (the first always applies their intellectual standards to everything; the second never does). What's invalid is when you use intellectual standards as a tool to get whatever you want, by applying the standards selectively.
  • Infinite Debt forges a middle path, praising Giving What We Can for telling people that you can just give 10% to charity and be an "Officially Recognized Good Person" -- you don't need to follow your principles all the way to giving away everything, or alternately, ignore your principles entirely. By following a simple collectively-chosen rule, you can avoid applying principles selectively in a self-serving way.
  • Bottomless Pits Of Suffering talks about the cases where utilitarianism becomes uncomfortable and it's tempting to ignore it.

But related ideas are in many other posts. It's a thread which runs throughout Scott's writing. (IMHO.)

This conflict is central to the human condition, or at least the WASP/WEIRD condition. I imagine most of Scott's readers felt similar conflicts around applying their philosophies in practice.

But this is really weird from a decision-theoretic perspective. An agent should be unsure of principles, not sure of principles but unsure about applying them. 

I get the impression that Scott implicitly believes maximizing his own values would be bad somehow.

Some of this makes sense from a Goodhart perspective. Any values you explicitly articulate are probably not your values. But I don't get the sense that this is what's going on in Scott's writing. For example, when he describes altruists selling all their worldly possessions, it doesn't sound like he intends it as an example of Goodhart; it sounds like he intends it as a legit example of altruists maximizing altruist values.

In contrast, blogs like Minding our way to the heavens give me more of a sense of pushing the envelope on everything; I associate it with ideas like:

  • If you aren't putting forth your full effort, it probably means this isn't your priority. Figure out whether it's worth doing at all, and if so, what the minimal level of effort to get what you want is. (Or, if it really is important, figure out what's stopping you from giving it your full effort.) You can always put forth your full effort at the meta-level of figuring out how much effort to put into which things.
  • If you repeatedly don't do things in line with your "values", you're probably wrong about what your values are; figure out what values you really care about, so that you can figure out how best to optimize those.
  • If you find that you're fighting yourself, figure out what the fight is about, and find a way to best satisfy the values that are in conflict.

In more SCC-like terms, it's like, if you're not a holy madman, you're not trying.

I'm not really pushing a particular side, here, I just think the dichotomy is interesting.


Arrow grid game

28 февраля, 2021 - 12:50
Published on February 28, 2021 9:50 AM GMT

There’s something I like about having different systems all the time. Apparently.


Privacy vs proof of character

28 февраля, 2021 - 05:03
Published on February 28, 2021 2:03 AM GMT

Below, I outline some musings I’ve produced while thinking about the private cryptocurrency Monero. I make no pretense to originality, cleverness, or exhaustiveness in covering considerations.

There are (at least) two distinct types of privacy. For instance, suppose you have an item in your house that you want nobody to see, even an intruder. One way you could handle this would be to buy a large black safe and hide the items in the safe, locked with a key that only you possess, and leave the safe on your desk. The other way would be to build a similar safe into a wall, and put a framed portrait in front of the safe. In the first case, an intruder can tell that they can’t open the black safe, but they know that you have something that you wish to keep hidden. In the second case, unless the intruder is perceptive enough to look behind the portrait, they will not even be able to tell that you have anything to hide. One could imagine an even better-concealed safe that would take painstaking effort to find - for the sake of this post, please imagine that it is very difficult to imagine that there might be anything on the wall behind a portrait.

You might prefer to own the second type of safe rather than the first, because the very fact that you possess something that you do not want others to see conveys almost as much as the sight of the item that you want to conceal. For instance, if you own something that proves that you violate certain social norms, the reason that you want to conceal it is to make others think that you actually abide by social norms - but if people only conceal things that do not abide by social norms, the fact that you are concealing something demonstrates that you do not, the very fact that you wanted others to remain ignorant of!

The hidden safe has another interesting property. If hidden safes exist, and it would be possible for you to install one in your home without others finding out, then it becomes extremely difficult for you to prove to others that you do not own a hidden safe. You could bring others inside your house, but in order to prove that you do not own a hidden safe you would have to take them thru a detailed enough level of search where they could find the hidden safe if it existed, which by hypothesis is no simple matter. Furthermore, in conducting such a search you would not be able to avoid parts of your walls that have embarrassing photographs of you as a child, while if you wanted to prove that you didn’t own a large black safe, you could cover up those photographs without any detriment.

This second property sometimes holds by design. For instance, take the case of the secret ballot. In many jurisdictions, it is not just possible to fill out a ballot in a booth where nobody else can see what you are doing, it is mandatory, and illegal to e.g. take photographs of your ballot to show others. This is precisely so that you cannot prove to others how you voted, so that votes cannot be bought or coerced.

The result of the second property is that it makes certain types of ‘proofs of character’ impossible. Consider the following scenarios:

  1. Alice is running for a position in the local branch of her preferred political party. Others suspect that she may actually be disloyal to the party. In order to increase people’s trust that she will act in the best interests of the party, Alice would like to prove that she voted for the party in the most recent election.
  2. I run a podcast where I interview AI alignment researchers about their papers, some of whom are personal friends of mine. People listening could be unsure that I’ve asked all relevant questions, and suspect that I have entered into an arrangement to ‘go soft’ on some interviewee in exchange for some kind of benefit. In order to demonstrate my integrity, I might like to show that no such conversations have ever taken place, and show the history of any financial dealings I have with interviewees.

If elections are run by secret ballots, then 1 is impossible. Regarding 2, if it is possible to have unrecorded conversations with others without special software, then I can’t prove that I have never had an untowards discussion with an interviewee. Furthermore, there is a cryptocurrency called Monero that has the property that nobody can tell who anybody is transacting with or how much they are transacting for (similarly to how cash works, and differently to how most blockchains work). There is currently a project called Kovri which aims to make it so that someone can access the Monero network without this fact becoming known by e.g. one’s internet service provider. As such, if Kovri were successful, as long as one could conceal a piece of paper containing one’s private keys, it would be a trivial matter to make a Monero transaction using Kovri and then delete the software used to do so from one’s computer, making it impossible to find out that the transaction ever took place or indeed that one ever used Monero without difficult forensic techniques. This means that I would be unable to prove that I have not received money from an interviewee without submitting to arduous and invasive scrutiny.

I regard it as somewhat unfortunate that this second type of privacy leads to the impossibility of proofs of character. An abating factor is that as far as I can tell, people very rarely seem to want to prove their character even when this is possible. For instance, one could film one’s whole life for several years before going into politics in order to prove virtue, quick-wittedness, and a lack of any ill-intent. These would seem to be desirable properties of leaders of countries, and yet I am unaware of any serious candidate doing such a thing. Perhaps a counterexample is the common custom of US presidential candidates to release their tax information before presidential elections, but this is hardly exhaustive, and might be rendered obsolete by the refusal of a recent presidential candidate to do so. My guess is that the lack of such proofs is caused by a combination of the poor character of such candidates relative to an ideal candidate, the paucity of scenarios where such a proof would be desired (either due to a lack of importance of such character, or a lack of relevant doubt), and a desire to not prove that one is extremely unusual.


Suspected reason that kids usually hate vegetables

28 февраля, 2021 - 02:52
Published on February 27, 2021 11:52 PM GMT

Here is a common vegetable preparation method in U.S. suburban homes.

Step 1: cut it up

Step 2: boil them in water until the flavor and texture is gone.

Step 3: Serve them without any kind of seasoning, or mixed with another food that could provide flavor and texture. Alternatively, put salt and pepper on them so that all vegetables just taste like salt and pepper.

Step 4: Tell the kids that eating a pile of bland mush with each meal is needed for being healthy.


In high school, me and several other people made the surprising discovery that if you eat vegetables raw, they actually taste alright, and some (like celery) even taste good enough to just snack on plain. Plus, there are things like stir-fry that make vegetables taste great.

I know this was common where I grew up. Do enough other people have similar experiences that raising knowledge of this is a great way to increase healthy eating?


Bootstrapped Alignment

27 февраля, 2021 - 18:46
Published on February 27, 2021 3:46 PM GMT

NB: I doubt any of this is very original. In fact, it's probably right there in the original Friendly AI writings and I've just forgotten where. Nonetheless, I think this is something worth exploring lest we lose sight of it.

Consider the following argument:

  1. Optimization unavoidably leads to Goodharting (as I like to say, Goodhart is robust)
    • This happens so long as we optimize (make choices) based on an observation, which we must do because that's just how the physics work.
    • We can at best make Goodhart effects happen slower, say by quantilization or satisficing.
  2. Attempts to build aligned AI that rely on optimizing for alignment will eventually fail to become or remain aligned due to Goodhart effects under sufficient optimization pressure.
  3. Thus the only way to build aligned AI that doesn't fail to become and stay aligned is to not rely on optimization to achieve alignment.

This means that, if you buy this argument, huge swaths of AI design space is off limits for building aligned AI, and means many proposals are, by this argument, doomed to fail. Some examples of such doomed approaches:

  • HCH
  • debate

So what options are left?

  • Don't build AI
    • The AI you don't build is vacuously aligned.
  • Friendly AI
    • AI that is aligned with humans right from the start because it was programmed to work that way.
    • (Yes I know "Friendly AI" is an antiquated term, but I don't know a better one to distinguish the idea of building AI that's aligned because it's programmed that way from other ways we might build aligned AI.)
  • Bootstrapped alignment
    • Build AI that is aligned via optimization that is not powerful enough or optimized (Goodharted) hard enough to cause existential catastrophe. Use this "weakly" aligned AI to build Friendly AI.

Not building AI is probably not a realistic option unless industrial civilization collapses. And so far we don't seem to be making progress on creating Friendly AI. That just leaves bootstrapping to alignment.

If I'm honest, I don't like it. I'd much rather have the guarantee of Friendly AI. Alas, if we don't know how to build it, and if we're in a race against folks who will build unaligned superintelligent AI if aligned AI is not created first, bootstrapping seems the only realistic option we have.

This puts me in a strange place with regards to how I think about things like HCH, debate, IRL, and CIRL. On the one hand, they might be ways to bootstrap to something that's aligned enough to use to build Friendly AI. On the other, they might overshoot in terms of capabilities, we probably wouldn't even realize we overshot, and then we suffer an existential catastrophe.

One way we might avoid this is by being more careful about how we frame attempts to build aligned AI and being clear if they are targeting "strong", perfect alignment like Friendly AI or "weak", optimization-based alignment like HCH. I think this would help us avoid confusion in a few places:

  • thinking work on weak alignment is actually work on strong alignment
  • forgetting work on weak alignment we meant to use to bootstrap to strong alignment is not itself a mechanism for strong alignment
  • thinking we're not making progress towards strong alignment because we're only making progress on weak alignment

It also seems like it would clear up some of the debates we fall into around various alignment techniques. Plenty of digital ink has been spilled trying to suss out if, say, debate would really give us alignment or if it's too dangerous to even attempt, and I think a lot of this could have been avoided if we thought of debate as a weak alignment techniques we might use to bootstrap strong alignment.

Hopefully this framing is useful. As I say, I don't think it's very original, and I think I've read a lot of this framing expressed in comments and buried in articles and posts, so hopefully it's boring rather than controversial. Despite this, I can't recall it being crisply laid out like above, and I think there's value in that.

Let me know what you think.


Participating in a Covid-19 Vaccine Trial #2: We pretty much knew it would work the whole time

27 февраля, 2021 - 03:47
Published on February 27, 2021 12:47 AM GMT

The past few weeks have been uneventful. I spend a few minutes each day filling out a questionnaire on the Patient Cloud app I installed during my first visit. The study is interested in reactions to the injection itself: redness, swelling, pain, etc. around the injection site, and the first section of the questionnaire checks for these symptoms. The next part is a self-screen for Covid-19 itself. Using the thermometer the nurses gave me, I take and report my temperature. There’s a checklist of symptoms to look out for: malaise, nausea, fatigue, etc. I’ve had nothing interesting to report, which I’m sure is a delight to the researchers.

I have my next injection appointment in about a week - I planned to write this second update sometime after that, since I would have a better idea of whether or not I received the placebo treatment. However, I got a very interesting email today (image below):

Dear PREVENT-19 Study Participant:

Novavax is happy to announce that thanks to many volunteers like you, we have completed enrollment in the PREVENT-19 study. We want to thank you for your commitment to participation as without it, we would not be able to progress this very important research into a new COVID-19 vaccine.

We also want to take the opportunity to notify you that the Institutional Review Board (IRB) has approved Novavax's plan for a blinded crossover. What this means is that all participants will eventually receive the real vaccine. For this to happen, we first need to demonstrate that the vaccine is safe and does prevent COVID-19 disease.  Participants who choose to remain in the study on their blinded treatment (sic). We anticipate that between one to two more months of follow-up will be required to reach that goal. Crossover vaccinations will be planned to follow thereafter.

We thank you for the extremely valuable contribution you are making to research and to society during these trying and challenging times.

Kind Regards,

The Novavax Team

My first response was a sense of relief that, worst case, I would likely receive the real vaccine within a few months. But then my critical thinking booted up. We are still experiencing a national (and global) shortage of vaccines - If the Novavax vaccines are believed safe and effective, shouldn’t they be going into the same supply of vaccines that everyone else is being served out of? I came up with two possible answers to this question. (Spoiler: both of them suck)

I am getting special priority because I volunteered to participate. If that is true, is that kind of kickback scheme remotely ethical? To participate in the trial, I needed to have the time, energy, and ability to sign up online, transport myself to the site, and then spend part of my day answering questions and letting the nurses poke and prod me a bit - all of which was made considerably easier by ability to speak English, access to a car, resources and free time, etc., in other words, exactly NOT the kinds of things according to which we should be allocating vaccines.

Novavax and the experimental team think it’s safe and effective enough to do the cross-over and give the control group the real vaccine, but the Very Serious People in charge of US vaccine approvals don’t. In that case, we’re just back to the regular story of an abundance of caution causing senseless morbidity and mortality; blah blah blah FDA Delenda Est.

Of course this is all occurring with the backdrop of the Novavax vaccine already having been tested with great results overseas, and oh my god why haven’t we been distributing this thing since last month?

If it turns out that I was randomized into the placebo group and they offer me the real treatment sometime in the future, I plan to take it despite not being particularly vulnerable - I don’t think me refusing on principle would result in someone else getting the dose any faster, but it might result in a lot of wasted effort contacting me repeatedly while the dose meant for me sits in a freezer unused.