Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 54 минуты 9 секунд назад

Towards a Bayesian model for Empirical Science

7 октября, 2021 - 08:38
Published on October 7, 2021 5:38 AM GMT

The aim of this post is to explore if we can create a model for doing Empirical Science that better incorporates Bayesian ideas.

The Current Model Of Doing Science

Imagine you are a Good Scientist. You know about p-hacking and the replication crisis. You want to follow all best practices. You want to be doing Good Science!

You're designing an experiment to detect if there's a correlation between two variables. For instance height and number of cigarettes smoked a day. You want to follow all best practices, so you write a procedure that looks something like this (taken from https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-control/):

  1. You find a large cohort of randomly chosen people. You use the SuperRandomizerV3 to teleport 10000 completely randomly chosen people into your laboratory, and refuse to let them out till they answer your questionnaire about their smoking habits, and allow you to measure their height.
  2. You consider a p value of 0.001 as significant.
  3. You calculate the correlation between their height and the number of cigarettes they smoke a day. By pure chance it's almost certain not to be 0.
  4. You calculate the the chance they would get this correlation assuming the true correlation was 0 (the p value).
  5. If p > 0.001 you conclude the experiment is a dud. No evidence of correlation. Better luck next time.
  6. If p < 0.001 you're golden! It's now been scientifically proven that height correlates with smoking habits.
  7. You preregister the experiment in detail, including the exact questions you ask, the p value you will consider significant, the calculations you will do, whether the test is two tailed or one tailed etc.

You then follow this procedure exactly, publish your results in a journal no matter what they show, and demand replication.

And that's pretty much the procedure for how to do good science under the current empirical model.

The great thing about this procedure is it doesn't require much subjective decision making. So long as your honest about it, and go carefully, you don't actually have to be very clever to do this - anyone who's capable of following steps and doing some simple statistics can carry out this procedure. It's reproducible, and the procedure can be easily checked for design flaws simply by reading it. This is important given how many people work in science - you want a technique that doesn't require brilliant practitioners, and is as externally verifiable as possible. 

The problem with the current model of doing science

I think, as ever, XKCD tends to sum it up pretty well:

  1. The procedure has no formal model for taking into account how likely the thing we're testing for is in the first place (priors).
  2. Significance thresholds seem a bit arbitrary. So 0.0009 is significant, but 0.0011 isn't. Why?
  3. There's no good model for combining multiple experiments on the same thing.

    On the one hand, if we'd done this experiment 1000 times and got p = 0.0011 each time, we would be dead certain this correlation was real. OTOH if we did the experiment 1000 times, and got p = 0.5 most of the time but 0.001 once we'd be pretty sure that the positive result was just a fluke. There are formal models for how to account for this if the experiments are exact duplicates, or completely unrelated, but not for experiments which are testing a similar but slightly different thing. For example see the second problem in this ACT post: https://astralcodexten.substack.com/p/two-unexpected-multiple-hypothesis


It doesn't tell me what the correlation is. It only tells me there is a correlation. I can calculate the most likely correlation given the data, but that's very unlikely to be the true correlation.

5. It looks at the chance that we see the data we see given the null hypothesis, but doesn't check how likely it is given the alternative hypothesis - only that it's more likely. For example I want to test the hypothesis that cranberries make people run faster. I pick a person at random, feed him some cranberries, and he beats the world 100m sprint record. The chance of this happening randomly are close to 0 - only 1 in 8 billion people are Usain Bolt, and he doesn't beat the world record very often either. So it looks like we've proved with absolute certainty that cranberries make you run faster.

However even assuming cranberries made people run faster, the chance that a random person given cranberries would beat the world record is tiny. Maybe, under H1, the top 10 sprinters in the world might have a decent chance at breaking the speed record - let's say 1/10. And under H0, they would have their standard chance of breaking the speed record - 1/1000. So my test only gives an effective p value of 1/100, not the naive value of 1/10,000 billion.

Doing BetterSplit Analysis and Research

Currently papers tend to have a conclusion. This might seem sensible, but it actually leads to problems.

Firstly, this encourages scientists to make the conclusion exciting. This creates all the incentives for p-hacking, and also makes it hard to publish negative results.

It also makes it easier for scientists to push an agenda, by representing their findings in a particular light, which may not always be the impression you would get if you actually read their experimental results. Similarly it's easy to find papers with the conclusion you want, instead of looking at meta-analyses which survey the whole field.

Finally in order to come to a conclusion via Bayesian updating, you first need a prior. The problem is that there's no practical objective way to come up with a prior (Solomonoff induction is not practical). This means that you can come to any conclusion you like based on the available evidence by choosing a suitable prior.

So I would suggest separating out reporting research, and analysis of the research.

The aim of the first is simply to present an experiment and it's results in a format which makes it easy to do Bayesian updating (more on that later). It should describe the experimental procedure, and the results, and may discuss possible con-founders, but should avoid any discussion of what the results mean.

Since all research papers are thus equally unexciting, (and equally important to Bayesian updating in the meta-analyses where the exciting stuff happens) there's much less incentive not to publish all experimental results, including those which find very little. in fact papers which find very little update the prior just as much as those which find a lot.

Meanwhile meta-analyses will try to come up with priors for particular questions of interest, and then sort through all the relevant papers on the topic, and apply the Bayesian update from these papers to the prior, possibly weighted on the quality of the paper/experiment.

Format of publishing experimental results

Trawling through the raw data of every single experiment would be far too much work for anyone doing a meta-analysis. If only there was some compact way of representing a Bayesian update, without knowing what the prior is first!

Fortunately there is!

First let's review Bayes formula:

P(A|B)=P(B|A)∗P(A)P(B).mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}

The probability of A given B occurred is equivalent to the probability of B occurring given A is true, multiplied by the prior probability of A, divided by the prior probability of B.

In other words to calculate  P(A|B), we multiply the prior by P(B|A)P(B). So it seems we can just calculate P(B|A)P(B) up front, report that in our paper and be done - all the meta-analaticians need to do now is a bit of multiplication!

Unfortunately that doesn't work - P(B) isn't a constant. If B is more likely given A, then as P(A) increases, P(B) does too, with limP(A)→1P(B)=P(B|A). So each time we update P(A) based on the results of 1 paper, we need to recalculate the multipliers for all the other papers before we can apply them to our adjusted priors. This is what stops us ever updating probabilities so much they become greater than 1.

However we can still fix this:

First we calculate  P(B|A) and P(B|¬A), and report these as R1 and R2 (short for Result1 and Result2).





This might not look much better but the great thing about it is it's a mechanical formula. A paper can publish R1 and R2 - what is the chance we would get the result if hypothesis X was true, vs if hypothesis X was not true. Then a meta analysis can take their previous estimate for how likely X is, plug these 3 numbers into a spreadsheet, and get a new updated likelihood for X, which they can plug into the results from the next paper. All without ever having to look at a the raw data from the paper.

In the real world things are a bit more complicated. It's very rare that hypotheses are discrete - usually I'm trying to find the value of some parameter X, where X can be any number in a range.

For example, in the experiment described at the beginning of this post (a correlation between height and number of cigarettes smoked a day), it's not a binary question (is there a correlation or not), but more a question of what that correlation is. The correlation coefficient can be any number between -1 and 1, but is usually close to 0.

Let's draw a graph. My prior for the correlation between these two variables might be something like this:

Prior for correlation between number of cigarettes smoked per day and height

Here the height on the graph represents a probability density, rather than a probability.

My data might look like this:


Scatter plot of cigarettes smoked per day vs height

Then I can graph P(B|A) - the probability that I would see the exact experimental results I got, given the value of the correlation. This will look something like this:

The probability of seeing the experimental results, given the correlation

Note that this time the graph peeks below 0 since the trend line is downwards sloping - we would be more likely to see our experimental results if the coefficient of correlation was negative.

(Now someone might point out that P(B|A) is always equal to 0 assuming the possible values of B are continuous. However the exact probabilities don't matter, all that matters is that the relative heights are correct, since when we use this graph we will normalize the results. So it's perfectly fine to use the probability density of seeing B given A instead of the probability.)

Bayes formula for probability densities is as follows: Let Pd(Ax) represent the probability density that A has value x.


Pd(Ax|B)=P(B|Ax)∗Pd(Ax)∫P(B|Ay)∗Pd(Ay) dy

This might look super complex, but it's actually extremely straightforward. Graphically we multiply out the two graphs - our prior and the chance of seeing the experimental results we saw, getting a result that looks this:

Our prior distribution multiplied by the chance of seeing our experimental results

At every single point we just multiplied the Y value on both graphs and plotted that as the new y value. As a result it's much steeper and more pointed.

We then simply scale the graph so that the total probability adds up to 1:

Same distribution, just scaled.

So all the paper needs to report for the meta analysis to be able to do it's Bayesian update is the probability distribution of seeing the results we saw, varying over whatever variables we may be interested in. Whilst finding a computable function to represent this distribution may not be possible, it should usually be doable to calculate this for a lot of dense points in the regions of interest, which can be published as a data set online.

So in place of a conclusion, each paper would basically just publish a graph saying: "Depending on what values X, Y and Z have, this is how likely it would be to end up with the results we saw. Make of that what you will."

Now this graph is always interesting unless it's a perfectly flat uniform distribution, since it will always adjust the prior. As a result there's very little reason for the scientists not to publish the experimental results whatever they find - there's no such thing as a negative result. Similarly, there's no P value, so there's nothing to hack.


We've looked at some of the problems with current methods of doing science, and suggested a methodology for improving science by focusing on Bayesian updates instead of p-values. 

We've looked at how we can publish a Bayesian update separate from any priors, and suggested that if papers were to publish their results in that format, it would allow meta-analyses to easily update on a paper, which would make a Bayesian methodology of Science more practical to carry out.

We've also suggested that if papers were to avoid drawing any conclusions at all beyond this, that would hopefully alleviate some other current problems with the scientific process today, such as P hacking and publication bias.

There's plenty more work that needs to be done to turn this into a practical methodology, but hopefully this is a good start!


Cheap food causes cooperative ethics

7 октября, 2021 - 04:52
Published on October 7, 2021 1:52 AM GMT

Nobody has ever fought a war over the oxygen in the air because there's plenty of oxygen to go around. We fight over things that are scarce and valuable. Historically, most wars have been fought over land and people. If you win you get to force your subjects to collect raw materials from the land. Sometimes these raw materials are mineral resources. Historically, most slaves have been forced to cultivate food.

It's hard to comprehend how important food staples used to be. In Edo Japan, wealth was measured in koku (石). One koku is (in theory) enough rice to feed one man for one year. The amount of koku a daimyo controlled was basically how many people he owned because a region's food staple production determined its carrying capacity and the human population grew until it hit carrying capacity. In other words, we bred until we were on the edge of starving to death. Most wars have ultimately been fought over land because land determines food production and food production was a matter of life and death.

My grandfather, who grew up in Taiwan before the green revolution, was too poor to afford rice. He ate sweet potatoes instead.

The green revolution of the 1950s and the 1960s increased food production faster than our population growth. On the Chinese version of TikTok there's a video of a guy eating a bowl of white rice with a spoonful of soy sauce. The comments are all of people feeling sorry for him. We have so much food these days that eating nothing but rice makes you surprisingly poor even by Chinese standards.

World War II ended in 1945—right before the green revolution. There has been no direct war between major world powers between then and now. I don't think this is a coincidence. Wars are usually about land and land is usually about food production and food production skyrocketed in the decades after World War II. (Birth control became widely adopted around the same time too.)

We still have small wars. We fight over oil and ideologies. But cars and capitalism aren't as important as food production. Fighting wars over food is stupid when food is cheap. Less fighting over food means less fighting overall. Countries being at peace with each other set the conditions for us to build more complex, interdependent trade networks. I think the idea that we're all human beings, regardless of our race, religion, sex and country of origin gained increasing power because cooperation is a winning strategy when there is enough to go around.

I predict that if per capita food production returns to the levels of 1914 then so will humankind's ethics.


Automated Fact Checking: A Look at the Field

7 октября, 2021 - 02:52
Published on October 6, 2021 11:52 PM GMT

Intro / Motivation

This post is the write-up of a few days reading the automated fact-checking literature. 

As AIs become more powerful, it would be very valuable to be able to train an agent to be able to tell us, not just whether particular statements are true, but why, and to explain their sources and reasoning as best as possible, especially in slow-takeoff scenarios. Fact-checking represents the present-day, practical end of the spectrum of possible work in this area, and I was curious what what had been done on it, the challenges the field faces, and whether it might shed light on the wider alignment problem.

Ultimately, I don't think there are any major positive surprises or insights generated by what I've read of the field. Instead the tale is more about the difficulties of getting high-quality data, and (especially given the current difficulty of the task) the need to carefully combine automated tools with natural human workflow in order to make something useful. Nonetheless, you might find value in some background on the field, its datasets and challenges.

If you're just interested in a general survey of the field, I would recommend reading the recent summary of the field Automated Fact-Checking for Assisting Human Fact-Checkers which talks through largely the same material from a more practical perspective. I'm instead looking more closely at the generation of the datasets and the techniques employed in solving them.


The few fact-checking datasets generally consist of tens to hundreds of thousands of sentences hand-curated by Mechanical Turk or similar workers who are told to create 'facts' from structured sources like Wikipedia. The core bottleneck is that generating explicit 'explanations' is both difficult and incredibly time-consuming at a size large enough to form a useful training set. 

There has been some movement towards the use of large, pre-trained models with the use of a fine-tuned BERT model, but there's not much visible activity to apply the big beasts of language modelling to integrated fact-checking systems (though work may well be ongoing at Facebook and elsewhere). These machine learning systems are therefore some way behind the cutting edge of language understanding and I expect large leaps would result from greater attention and firepower. At the moment, state-of-the-art fact checking systems are piecemeal and machine learning approaches are still sometimes outperformed by much simpler statistical algorithms. Fully automated approaches have been unsuccessful

The primary practical use of fact-checking systems thus far has been in collating facts worth checking, and extracting an initial set of relevant pages (a task also carried out by internet search). The task of actually bringing specific pieces of evidence to bear on a claim and rendering a judgement is only in its infancy.

Wikipedia represents probably the largest and highest quality dataset with millions of articles leveraging both internal and external links to substantiate the points made, and it looks likely that this will form the bulk of future training for good explanation, alongside more scarce but higher quality human feedback.


Creating large datasets to train an ML-based fact checking system is perhaps the biggest obstacle to a useful system. Here I go over what seem to be the major datasets and how they structure and source their data.

Fact Extraction and Verification (FEVER)

FEVER (paper, dataset website) is the most commonly used fact-checking corpus over the last few years. Its primary virtue is its large size and that it has a large set of claims for which the 'supporting evidence' sentences are explicitly picked out, whereas other datasets often only contain True/False/Not Supported labels.

They take the introduction of 5,000 of the most popular Wikipedia pages, and the pages that are linked from these introductions to get a set of 50,000 linked introductions. Facts are generated by hand-writing sentences labelled either Supported, Refuted, or Not Enough Evidence. For the former two classes, the writer also label the sentences that contribute to the statement being supported or refuted.

They generate 185k claims, of which about 135k have supporting sentences. The structure of the dataset is a separate entry for each supporting sentence, with each entry having the claim, the label, the title of the supporting page and the sentence ID of the supporting/refuting sentence. 

Example showing structure of the dataset. Indicates that the claim is supported by (0-indexed) sentences 0, 1 and 6 of the page 'Oliver Reed', and the first sentences of the introduction for the films 'Castaway' and 'Gladiator', referenced in the 3rd and 4th sentences of the introduction respectively.

They used a group of 50 native English speakers but appeared to have significant difficulty getting their labellers to do more than trivial rewordings or negations of sentences in the introduction, designing the interface to have suggestions for other types of mutation and an 'ontology diagram', as well as highlighting negations to discourage their overuse.

Browsing the dataset shows that the claims are largely very simple and often of quite low quality.. The first claims in the Supporting/Refuting/Not enough evidence categories are, respectively, 'Nikolaj Coster-Waldau worked with the Fox Broadcasting Company.', 'Adrienne Bailon is an accountant.' and 'System of a Down briefly disbanded in limbo.'.


This is another dataset (paper, GitHub for data) that uses Wikipedia as its ground truth but it sources its supporting evidence from PDFs cited as sources (as these are judged to generally be of a higher standard). It takes sentences from Wikipedia articles which are immediately followed by citation of a 'well-formed' document (mid-length pdf, in English). They use MTurk workers to generate negations and mutations of these sentences. 

The dataset itself simply consists of the claim, whether it is True or False, and the URL of the PDF with supporting evidence, along with a context field which is the preceding sentences up to the last paragraph break. There is no 'correct' supporting claim - instead the detection of particular supporting sentences is an (optional) part of the process of determining if the claim is true or false.

This level of technical language is common to most of the entries in the database, giving a high degree of difficulty to pull it out of a document which may have many other themes.

This is clearly a much more ambitious challenge, and one which, relying on humans only for mutations and not for the sentences/sources themselves, could scale up to a much larger sourcing and entailment dataset. 

The authors' own approach to solving their own challenge is to extract the most similar claim by picking the sentence with the lower normalized Levenshtein distance and then using either a linear classifier based on a hand-crafted features, or a neural network model trained on the SNLI (see below).


This dataset (arXiv, dataset) tries to teach structured data by taking tables from Wikipedia (which is a dataset in itself) and asking MTurk users to create statements which are either entailed or refuted by the table.

The use of MTurk workers trying to create data as quickly as possible and without much training is clear. Poor quality English is very common. The meaning is usually, but not always, clear. They apparently restricted their data gathering to English-native countries and checked one statement from each task that it met criteria which include grammar, but it seems they've had to be quite generous in allowing poor grammar in their sentence creation.

This is the first example from TabFact I came across, and is broadly representative of the datasetStanford Natural Language Inference Corpus (SNLI)

This isn't exactly a fact-checking dataset but rather the preferred dataset for learning logical entailment: it asks the solver to decide whether a given 'hypothesis' is an entailment of, a contradiction of, or is neutral with respect to, the 'text'. 

Examples from the SNLI. 

A state-of-the-art fact-checking system generally is not a single system - instead it consists of separate models which A. pick out relevant documents, B. select relevant sentences from these documents, and C. decide whether these sentences support or refute the statement under question. 

The SNLI is widely used to solve step C of the above process, either as the primary training data, or as a source of pre-training for models which are then trained on FEVER (which also trains some degree of sentence selection but only offers paragraph-sized example documents).

Other datasets of interest

The fact checks of Politifact and others have been collated here. 

The Fake News Challenge has a small but reasonably high quality dataset assessing whether sentences agree with, disagree with, or are not relevant to a claim.


Here's a summary of the main themes that came up when reading how researchers have tried to solve these datasets, and the automatic fact-checking problem more broadly.


There has generally not been a single model which attempts to solve the full stack of challenges. Instead, a pipeline of different models is used to get a complete system. The core of this system is Document Retrieval -> Sentence Selection -> Recognizing Textual Entailment and there is generally not an attempt to combine these into an end-to-end-trainable system. Instead they remain as separately trained units where the intermediate outputs can be inspected and used as outputs in their own right. 

For sentence selection and recognizing textual entailment, the datasets above seem to be the major sources of training data. For evidence retrieval, I've not looked as closely, but there are some very different approaches. The human-assistance tools seem to rely on taking the top results from Google as a starting point and then filter the documents by a few, mostly hard-coded criteria. 

The more ambitious pipeline approach of Nie at al instead use a neural document matcher which first narrows down relevant documents by simple keyword matching (I think using Wikipedia as its document pool), and then uses the FEVER dataset to train an LSTM-based ANN to learn whether there is a high relatedness, by using the sentences in FEVER listed as evidence for the claim as the positive cases of relatedness. (I think this is an accurate summary.. I found their methods section rather lacking when it came to how the model was trained.)

Helper Systems

Current systems are not sufficiently competent to be of much use on their own, so if models are intended to be of practical use then they come integrated in programs which assist a potential fact-checker. 

Ideally I'd be able to explore these in more detail but I've not been able to test these systems. The two best documented systems are FullFact's system and the BRENDA system, which used to be publicly available but has now been folded into the proprietary Factiverse demo.  I've requested trial access to both but so far without success.

All other tools that I've found attempt the more limited task of matching claims found online and in speech to previously checked claims. While a less ambitious project, a hypothetical version of this system which was able to point to the best explainer of the background of any claim would be a very valuable human-augmenting tool so it's worth keeping an eye on.

However, even these systems have struggled, as you can see from this rather sheepish article about how difficult it has been to connect their database of fact checkers to a real time audio feed from live debates.

Facebook has a predictably more upbeat piece about its own efforts which work towards identifying whether the content of some text or image is functionally identical to anything which has previously been checked but I don't think it has any publicly documented attempts to determine the truth value for itself.

Large Text Models

As many of you will know, perhaps the biggest ML development in the last few years has the power of huge, transformer-based language models to understand text at a fundamental level than had previously been possible. These have become state-of-the-art in a number of domains and I would expect that any highly successful system would build upon something of this nature in order to extract the most information from the text received.

As far as I can tell, the most advanced text-model applied to the problem is a 110M parameter version of BERT used by the authors of TabFact (Chen et al) and fine tuned to learn whether statements were entailed by Wikipedia tables, training for another 10k steps. This is obviously a step in the direction of big models but is a long way behind the state of the art (in terms of parameters, this is ~1000x smaller than GPT-3, for example) and this is reflected in the poor performance of these models, only working out if the relatively simple statements were true or false in about 70% of cases.

Instead, this smaller BERT model was marginally outperformed by a much more bespoke system for which they created a domain specific language to compose table-actions into a boolean result, converted these programs to strings, and then used a transformer to learn whether a given program string learns the T/F value of a statement.

Also interesting was the difficulty in linearising these kind of tables. At least with the limited fine tuning that they applied, the BERT model wasn't able to perform at all well when given the table in its original structure. Instead, they used a 'template' method where they would feed in sentences of the form 'row $ROW the $PROPERTY1 is $VALUE1'. With this more natural sentence structure, the model was then able to at least match the program-generation approach described above.

I expect that if someone were to apply the techniques from Learning to Summarize from Human Feedback - that is, fine-tuning a 100B+ parameter model on limited but genuinely high quality data and using high-concentration human ratings of success to drive maximizing behaviour then this would easily outperform anything that the field has produced so far.


Ultimately I see this as an area which is still struggling to find its feet, both because of the difficulty of the problem and the difficulty of getting high-quality data to train any system. There's a lot of untapped potential for the use of large language models and investment in good quality feedback - I think that the first somewhat useful system is probably 2-5 years away, depending some specialized investment from but offers a promising use case for human-machine co-working to flourish. Unfortunately, the issue of how to create a system which aggressively tries to find the truth, rather than replicating what is expected from a human checker, remains untouched by current work.


Considerations on Compensation

7 октября, 2021 - 02:37
Published on October 6, 2021 11:37 PM GMT

This post was prompted by the recent announcement of Lightcone Infrastructure, and particularly Elizabeth's response in a comment thread on compensation.  I'd like to thank Max Wallace, Ruby, and Miranda Dixon-Luinenburg for their comments and copyediting.

Context (& disclaimer): I'm a software engineer and donate ~exclusively to longterm-focused orgs (currently MIRI).  While most of the arguments are intended to be broadly applicable, this post was written with that specific framing in mind, and one argument does explicitly depend on the question of AGI timelines.

You're an EA-aligned organization that isn't funding constrained.  Why should you (or shouldn't you) pay market rate?

Talent Pool

The econ 101 view says that if you want more of something, one easy way to get it is to pay more money.  But where exactly would we be getting "more" from?

The already-convinced

There is a group of people who already know about your mission and agree with it, but aren't doing direct work.  This group consists of:

  • Those earning to give
  • Those who are earning enough to give but don't (for a variety of reasons)
  • Those who don't have the capacity do either direct work or earning to give

On the margin, who in this group could be convinced to do direct work?

Those earning to give

There are many reasons why someone might be earning to give instead of doing direct work, assuming they have the skillset necessary to do the second (or can acquire it).  Here is a list of plausible reasons that could be overcome with more money.

  • Information -  they may simply not know that their favorite EA org pays (close to) market rates!  This is an area where there may be some "free wins" obtainable with just better PR - in that sense I think Lightcone's announcement and public salary listing is a great start.
  • Maintaining their present lifestyle - this one is pretty sensitive to the actual numbers involved, but it needs to be said that in the Bay Area there are things that are difficult to afford on typical non-profit salaries, like "owning a house within reasonable commuting distance of the office" and "having kids".  These can be concerns even for e.g. highly-paid software engineers, if they've already done one or the other (or both!) and thus locked themselves in to those costs.
  • Maintaining optionality - if you take a pay cut to do direct work and circumstances change in the future such that it no longer makes sense for you to keep doing whatever you're doing, you have less money and thus fewer options to deal with whatever problems may arise.  This doesn't matter if you're operating on short timelines and are extremely confident in them.  Kinda sucks if you're only mostly sure, though.
  • Hedging against the loss of career capital - technically this belongs under "Maintaining optionality" but it's a fairly specific point that I don't think I've seen raised elsewhere so I wanted to emphasize it.  At least in software, most work that you'd do at an EA org will not grow your career capital, compared to the counterfactual work you'd be doing at a better-paying tech company.  (The major exceptions to this are roles at research orgs which are doing large-scale ML R&D, which seem extremely similar to such roles at tech companies and look just as good if not better on your resume.  Think OpenAI, Anthropic, etc.)  If you jump in the pool, you'll likely have a harder time getting out, later, and this gets worse the longer you stay in.  For a motivated and skilled software engineer, this could easily dwarf the explicit pay cut when it comes to calculating foregone future earnings.
  • Social expectations - this is mostly just here for completeness' sake.  I don't expect there are a huge number of people who would go do direct work except that their family/friends/social circle would look down on them for taking a pay cut.  It's a big world out there, but I'd expect subtler status concerns to dominate and those are harder to move just by throwing money at them (though see the point about PR under "Information").
  • Preference for industry work - some people might expect that they'd enjoy direct work less.  Maybe they'd be happy to either take a pay cut, or sacrifice some of the intrinsic pleasure of the job, but not both?
Those who are earning enough to give but don't

If we want to be precise, there's obviously a spectrum - many people who are earning to give could probably give more without sacrificing anything, except optionality.  This group is all the way on one side of the spectrum.  When I was in this group the loss of optionality seemed to be the dominant factor informing that decision.  I don't have great insight into what else could be driving that for other people, though, beyond those factors also listed in the previous category.

Those who don't have the capacity do either direct work or earning to give

If the lack of capacity is a permanent condition, more money obviously won't do anything, so we can ignore that.  If it's a temporary condition, like "still a university student", then there are probably some cases where "more money" could be motivating in the right direction (focusing on things more applicable to direct work, graduating faster, dropping out, etc.).  I think the effect here is pretty marginal, though.

The skeptical

I don't think more money moves the needle for anyone who's familiar with a cause area but isn't convinced it's worth working on or directing resources to.

The not-yet-aware

Most (extremely online) software engineers have never heard of EA, but they have heard of FAANG[1]!  I think it's extremely likely that the number of software engineers who would be mission-aligned if they but knew about the mission is substantial and each additional engineer that first hears about [insert EA org here] on an online tech forum because "[insert EA org here] is competitive with FAANG, isn't that crazy??" is another opportunity to introduce someone to the community.  PR concerns deserve their own section but are not entirely negative.  If we are substantially bottlenecked on talent (particularly experienced talent), then increasing the top of the funnel is an urgent priority and could be much more effective than trying to pull from the relatively small set of people who are already aligned on mission, have the necessary skillset & talent, and aren't otherwise constrained by exogenous factors.

One potential downside here is that the pipeline could fill up with technically qualified but unaligned (or worse, pretending-to-be-aligned) candidates.  There are ways to manage this but it does deserve some thought.


One obvious concern with paying top-of-market salaries is the PR risk.  "Local non-profit pays its engineers half a million dollars a year!" certainly looks like a bad news headline, but I think this is not actually a huge concern.

First, "Local non-profit pays its engineers 300k per year!" does not have a meaningfully different effect on the impression the median headline-reader comes away with.

Second, my subjective impression (which could stand to be validated empirically) is that most sources of funding for orgs focused on long term causes and meta work won't be offended by paying market-rate compensation for talent.  Quite possibly they would have the opposite reaction!  There is a real risk here that it could turn away some potential future donors (or even employees) on the margin, particularly those who aren't yet familiar with EA and the associated goals and principles behind it.  If you strongly think we live in a universe with a short timeline (<30 years) to AGI, this is not a significant factor.  If not, this deserves consideration.

Other Considerations

Let's imagine you're spinning up a new EA organization and you have more money than you know what to do with.  You have so much money that you've started paying people $500 to write book reviews!  (I kid, I kid.  $500 book reviews are cheap, as far as a hits-based approach goes.  But they also aren't something that MIRI/LW/Lightcone had money for 5 years ago.)

How do you decide what you're going to pay people?  I'm not totally sure what strategy I would try first, but I don't think it's "try to capture as much surplus 'value' as I can from the relationship".  I wouldn't want to optimize for being just barely the best option my desired candidates have available to them - I'm in a rush!  I'm under a time crunch!  I need as many qualified candidates as possible beating down the doors to work with me.  I want to be solving the problems of "how do I filter for the best of the best" and "how do I grow an organization as fast as possible while remaining strongly mission-aligned", not "where on earth am I going to find motivated, competent, and aligned engineers to work early-stage-startup-style overtime on what is effectively line-of-business software"!

That's just a guess as to what I'd come up with if I was trying to write a comp policy from first-principles, anyways.  I'm not an HR professional and there are certainly considerations I could be missing, but even accounting for things you can't put in writing I don't think the case for paying below-market rates is terribly motivating.

  1. FAANG stands for "Facebook, Amazon, Apple, Netflix, and Google".  Originally the acronym was FANG, and was used to refer to a group of high-performing tech stocks, but was picked up by software engineers online to talk about tech companies that were known to pay notoriously well at the time. ↩︎


How do bounded utility functions work if you are uncertain how close to the bound your utility is?

7 октября, 2021 - 00:31
Published on October 6, 2021 9:31 PM GMT

If you are trying to calculate the value of a choice using a bounded utility function, how can you be sure whether you are close or far from the bound, whatever the bound is? How do you account for uncertainty about how much utility you already have? Does this question actually make sense?

Recently I have come across arguments against using a bounded utility function to avoid Pascal’s Mugging and similar “fanaticism” problems.  These arguments, such as Section 6 of Hayden Wilkinson’s paper “In Defense of Fanaticism” and the Less Wrong post “Pascal's Mugging for bounded utility functions” both use a novel argument against bounded utility functions. If I understand them correctly, they argue that bounded utility functions cannot work because it is impossible to know how much utility one already has. This means one cannot know how close to the bound their utility is, and therefore one can never how much to discount future utility by.

Wilkinson’s paper uses the example of someone with an altruistic bounded utility function that is essentially total utilitarianism.  So they want to increase the total utility of the universe and, because they have a bounded utility function, the value of additional total utility decreases as it approaches some upper bound. If I understand his argument correctly, he is saying that because this agent has a bounded utility function, they cannot calculate how good an action is without knowing lots of details about past events that their actions cannot effect.  Otherwise, how will they know how close they are to the upper bound? 

Wilkinson analogizes this to the “Egyptology” objection to average utilitarianism, where an average utilitarian is compelled to study how happy the Ancient Egyptians were before having children.  Otherwise, they cannot know if having children increases or decreases averages utility. Similarly, Wilkinson argues that a total utilitarian with a bounded utility function is compelled to study Ancient Egypt in order to know how close to the bound the total utility of the world is.  This seems implausible, even if information about Ancient Egypt was easy to come by, it seems counterintuitive that it is relevant to what you should do today.

“Pascal's Mugging for bounded utility functions” by Benya introduces a related problem. In this scenario, a person with a bounded utility function has lived an immensely long time in a vast utopia. Because of this, their utility level is very close to the upper bound of their bounded utility function. Pascal’s Mugger approaches them and tells them that all their memories of this utopia are fake and that they have lived for a much shorter time than they believed they had.  The mugger then offers to massively extend their lifespan for $5. The idea is that by creating uncertainty about whether their utility is approaching the bound or not, the mugger can get around the bounded utility function that normally protects from mugging.

One way around this dilemma that seems attractive to me is to use some version of Marc Colyvan’s Relative Expected Value theory. This theory, when looking at two options, compares the differences in utility, rather than the total utility of each option. This would seem to defeat the Egyptology objection, if you cannot change how much utility the events in Ancient Egypt were worth, then you don’t factor them into your calculations when considering how close you are to the bound. Similarly, when facing Pascal’s Mugger in the far future, the person does not need to include all their past utility when considering how to respond to the mugger.  There may be other approaches like this that discount utility that is unaffected in either choice, I am not sure what the best formulation would be.

However, I am worried that this approach might result in problems with transitivity, or change the ranking of values based on how they are bundled.  For example, if an agent with a bounded utility function using Relative Expected Value theory was given offers to play a lottery for $x 1,000 times they might take it each time.  However, they might not pay a thousand times as much to enter a lottery for $1,000x.  Am I mistaken, or is there a way to calibrate or refine this theory to avoid this transitivity problem?

I would love it if someone had an ideas on this topic. I am very confused and do not know if this is a serious problem or if I am just missing something important about how expected utility theory works.


2021 Darwin Game - Everywhere Else

6 октября, 2021 - 23:39
Published on October 6, 2021 8:39 PM GMT


The Shore is an inhospitable wasteland. Algae is available, but it's not very nitrutious. Coconuts offer an even worse calories-to-digestion ratio.

Name Carrion Leaves Grass Seeds Detritus Coconuts Algae Lichen Shore 0 0 0 0 20 1000 1000 10

No coconut eaters got established in the Human Garbage Dump, which means no coconut eaters got established in the River or Shore instead. The only available food is algae. The Shore is basically a small bit of Ocean biome with ⅒ of the algae. The algae-eating winner of our Ocean competition has zero speed so it cannot migrate to the Shore.

To top all of it off, predators migrate in from both sea and land.

Soonbegons migrate from time to time to eat the Detritus. Otherwise, the Shore is mostly a graveyard.


Soonbegons (kind of) by Martin Randall.


The grassland has lots of grass and even more seeds.

Name Carrion Leaves Grass Seeds Detritus Coconuts Algae Lichen Grassland 0 100 1000 2,000 0 0 0 50

The Grassland took 500 turns to establish an equilibrium.

Generations 1-500

Goes Extinct in Generation Species 11 Meercat colony 14 Flock of birbs 15 Bool 16 Big Oof 17 Empowered Turtle 17 glpp511 18 cg-bird 19 Nyarlathotep 20 Human 21 Tiny Snek 21 grass_mouse 22 Sleepypotat 22 Frontier-W8 23 Armored Nutcracker 23 Toxikeet 23 lGha-S541 24 Cowlagor 27 Karthosorox 30 Piranhakeet 30 cg-fastbird 31 Wampirek 32 Yonge_Snake 34 Sheep 37 Jackrabbit 39 Grassland Tribble 39 Grassland Aphid 40 basic seed fodder 40 Galumphers 40 lGca-AS154 41 GrassSeeater 41 Small Moth 41 SpeedyLichen 41 BeaupreyButGrassland 41 Tribble 42 DefinitelyJustARock 44 Weedle 45 Bob 45 SeedyEaty 45 Locust-Seeds 47 Tribble Hunter Hunter 49 Goat 51 seed fodder 52 Seed Eater Eater Eater 53 Frontier-W2 56 Trash Panda1 56 mediocre 1-2-s 59 Common Rat 61 Omnivorous Tribble Hunter 62 Frontier-W5 63 Siolid 65 Cannibal Locust 68 Tribble Hunter Hunter Hunter Hunter 73 Ziarnojad Malutki 74 Horned Owl 78 Yonge_Omnivore 83 Medium Seedrat 86 Maverick Goose 89 Squish 187 Mooshroom 294 Killer Bunny 345 Untielopay 423 6-0-3 seed 466 Yonge_Vegan Generations 500-2000

The grassland establishes an equilibrium. The only native species left is the Hopsplitter.

You can't see it but there's a single Venomoth on the bottom of the graph.

Winners Species Native Biome Venom? Weapons Armor Speed Forage? Temperature Adaptation? Creator Social Media mediocre 1-3-s Temperate Forest Neither 1 0 3 Seeds Corm None Hopsplitter Grassland Neither 0 0 10 Grass;Seeds Nem None Brown Bear Temperate Forest Neither 3 0 6 elspood None Venomous Snark Desert Venom + Antivenom 1 0 3 Carrion Heat (Allows survival in the Desert) DaemonicSigil Twitter Venomoth Rainforest Venom + Antivenom 0 0 3 Grass aphyer None Temperate Forest

The temperate forest has lots of leaves and significant seeds.

Name Carrion Leaves Grass Seeds Detritus Coconuts Algae Lichen Temperate Forest 0 2,000 100 1000 0 0 0 50 Generations 1-200

The population crows, crashes, and then grows again.

Goes Extinct in Generation Species 9 Grizzly Bear 12 Optimistic Omnivore 13 LeavySpeedyTanky 13 The Dark Secret of Harvest Home 14 Colonoscopies 15 TFP511 15 Gobbledygook 16 Boojum 16 Ankylosaurus 16 Monstrous Bandersnatch 17 Arboreal Assailant 17 Ultimate Lifeform 17 Bear2 18 BeauOmni2 21 FastHerbivore 22 RockMonkey 22 Gino Soupprayen 23 Hedgie 24 Basilisk 24 Meta-Lichen 25 Armored Pigeon 26 Lichen Bug 26 Fox 26 Predator 29 Forest Leaf Blight 29 Leaf Tribble 31 TreeSeeater 31 Yonge_Slug 32 Gipgip 32 You Worm! 33 Leafy Luncheoner 33 Caterpie 33 Spider 34 Forest Dragon 34 Woodland Locust 34 Forest Tribble2 40 Munchers 41 Skitter Critter 52 SpeedySeedy 57 Unicorn 57 mediocre 1-3-s 57 lF-LM732 60 Songalagala 62 Mango 69 Sky Shark 70 Locust Eater Eater 71 Brown Bear 72 Snake 78 Bullfrog 79 Pidgeotto 81 Fast Venomous Snake 82 Morpork Owl 85 Deadly Mickey K 89 lFa-G172 93 Phanpy 97 Bear1 104 Omnom 109 Ken Nishimura 110 Forest Scorpion 114 Yonge_Defense 140 Snark 152 Bastion 169 Titanosaur Generations 200-2000

The system achieves stability stable (except for a random walk among equals).

Goes Extinct in Generation Species 219 Gypsy Moth 272 Armored Sloth 290 Donphan 333 Armadillo1 336 Szaromysik 342 Turtling Trencherman 762 Bofa #0461-B Winners

None of our final species have venom or antivenom.

Species Native Biome Venom? Weapons Armor Speed Forage? Creator Social Media Cutiefly Temperate Forest Neither 0 0 0 Seeds alkjash None Brown Bear Temperate Forest Neither 3 0 6 elspood None Forest Finch Rainforest Neither 0 0 3 Leaves MA None Snark Temperate Forest Neither 0 0 10 Leaves Vanessa None Forest Tribble1 Temperate Forest Neither 0 0 0 Seeds simon None Mutant Two-Headed Speed Snail Human Garbage Dump Neither 0 0 2 Grass DaemonicSigil Twitter

An honorable mention goes to the N054J's Forest Tribble2 which is identical to the Cutiefly and the Forest Tribble1.


The Rainforest has lots of leaves and grass but few seeds.

Name Carrion Leaves Grass Seeds Detritus Coconuts Algae Lichen Rainforest 0 1000 2,000 100 0 0 0 50 Generations 0-1000

It takes 1000 generations to establish an introduction.

Goes Extinct in Generation Species 11 Quartosaurus 11 Jon Rahoi 12 Stinging ant colony 14 Slug 16 Smok Wawelski Przezuwacz 18 Tree Clam 19 Slebbon 22 Rainmun 25 J's RLB 25 luden 29 Rainforest Leaf Blight 31 Tinysnek 32 Agyneta insolita 36 Crocodile 37 Giant Snake 39 Forest Ape 40 Very Lesser Forest Dragon 42 Birgus 47 Deerling 49 Elephant 60 lRa-G256 61 Murder Hornet 64 Yonge_Rainforest 66 BeauOmni2ButRainforest 79 Jungle Hare 81 Stomporz 84 Raincow 90 Grazing Snake 91 Wombat 93 basic grass fodder 95 Armadillo2 98 Rainforest Aphid 100 GrasseatyTank 111 Bofa 0461 119 Arboreal Grazer 127 Boop 128 Nine-Banded Armadillo 145 Tell Masoud 156 Rain Tribble 168 Lily the Unicorn 187 Panther 283 Forest Finch 305 mediocre 1-3-g 306 Will Die 316 lRa-GLM281 318 Fodder grass 344 Leaf Truck 537 Cheetah Generations 1000-9000

The population oscillates from there. A small number of Snarks and Brown Bears are not visible.

Goes Extinct in Generation Species 1485 Rony Winners Species Native Biome Venom? Weapons Armor Speed Forage? Temperature Adaptation? Creator Venomoth Rainforest Venom + Antivenom 0 0 3 Grass aphyer LeavyTanky Rainforest Antivenom only 0 10 0 Leaves ViktorThink Untielopay Grassland Neither 2 0 10 Grass Heat (Allows survival in the Desert) Anonymous #5 mediocre 1-3-s Temperate Forest Neither 1 0 3 Seeds Corm Forest Finch Rainforest Neither 0 0 3 Leaves MA Snark Temperate Forest Neither 0 0 10 Leaves Vanessa


Cafe Meetup: Debugging Misconceptions

6 октября, 2021 - 22:28
Published on October 6, 2021 7:28 PM GMT

Note: 1:30pm is just the gathering time, main topic is not until 2:30pm.

For this Saturday's topic, we're going to share our experience in "debugging" others' misconceptions -- cases where they made an incorrect inference about how to fix a problem, and how to trace back to the root cause. A (relative) newcomer, Joshua, mentioned his experience in handling support bugs, where he had to do that a lot.

Joshua will kick us off by sharing his experience in that role, and then we can take turns discussing our own techniques, and past situations with identifying root causes of misconceptions.

Looking forward to seeing those who attended the ACX Everywhere meetup! (This is in about the same place as that, except inside the store's cafe instead of the park area.)

To find us, look for the LW and SSC signs. People start arriving at 1:30 pm, and the main activity will begin at 2:30 pm. In light of the developments about Covid cases in Austin, please follow store guidance on masks.


Is GPT-3 is already sample-efficient?

6 октября, 2021 - 16:38
Published on October 6, 2021 1:38 PM GMT

(Concrete, easy-to-answer question below, explanation first)

Common adage: Modern deep learning techniques are sample-inefficient; it takes loads of data for them to learn things. If you pre-train them, it takes less additional data for them to learn something new, but still compared to humans it takes a lot.

Elsewhere, based on papers like this and this, various people have extrapolated the following takes:

--It seems like bigger neural nets need to see less data to reach the same level of performance.

--It seems like bigger neural nets need fewer epochs to reach convergence. Soon they'll only need to see each data point once. (Search this for "multiple epochs")

I feel like this take is in tension with the common adage. I wonder: If there is a fact mentioned in GPT-3's training data, how many times does it need to be mentioned before GPT-3 comes to know that fact? For example, I'm told that GPT-3 knows the names of most prominent members of the rationalist community. How many times has it seen each name? Are we talking ten times, or ten thousand?*

I'd be interested to hear people do a bit of a search for the "most sample-efficient/obscure fact" in GPT-3's repertoire. In this manner we could quantity how many times GPT-3 needs to see something before it learns it. (Maybe we don't have access to the dataset used to train GPT-3. But people at Eleuther.ai have The Pile, right? And they've trained big transformers on it? We could answer the question easily and precisely there, no?)

Or am I thinking about this all wrong somehow? This seems like an obvious idea, I wonder why I haven't heard of it before.

*Suppose it is ten thousand. Then that means one in every ten million two-word strings on the internet is "Paul Christiano." (The dataset for GPT-3 was 300B tokens) Add in all the other rationalists/EAs and probably it means one in every hundred thousand words is the name of some prominent rationalist/EA. Surely this is too much, no? It seems way too much according to Google Ngram Viewer.


Theory: The Political Incentives for Progressive Policy Promote Policy Failure

6 октября, 2021 - 15:10
Published on October 6, 2021 12:10 PM GMT

Thanks to Miranda for providing feedback on drafts and proofreading via the proofreading service. 

I think there is an incentive problem in Leftist and progressive politics (and therefore democracy more broadly) that isn’t adequately discussed. The goal of progressive politics is said to be helping people who have been disenfranchised unfairly. Naturally, progressive policies and institutions draw much of their support from disenfranchised people. However, what happens to the base of support for these institutions and policies (which can be thought of as alive mimetically) if they actually succeed at helping people? With less disenfranchisement, one would imagine there would be less support for progressivism. Thus, progressivism may be operating under a perverse incentive structure where it is rewarded for creating disenfranchisement in society.

Things which are known to shift one’s attitudes away from progressivism and towards conservativism are getting married and having kids. One would also imagine that welfare programs would have higher support among those who use them. People getting married, having families and not needing welfare are things which are broadly considered to be good. Yet, if Leftist policies were to succeed at moving these metrics in the right direction (more marriage, more kids, less welfare recipients), they would push the policy preferences of their country to the Right.

One might argue that as the Right is incentivized to increase these things (to push people’s policy preferences to the Right), there is a fair balance in the system. However, I think this is wrong due to the asymmetric nature of creation and destruction. It is far easier and less energy intensive for the Left to implement entropic destructive policies than for the Right to implement extropic creative policies. Thus, the Left has an easy mechanism to strengthen its movement (enhancing disenfranchisement) while the Right has no similar easy mechanism.

This does not require any conscious malice by the Left. All that is required is for the Left to implement policies which fail or backfire, and then never fix them because they are rewarded electorally for the failure, rather than punished. The Right, on the other hand, needs to boost employment, wages, savings, decrease house and family starting costs and increase marriage rates in an environment where they are only in power half the time and their efforts can be antagonized by the Left all the time. Clearly, this is a very asymmetric competition. The Right rarely attempts those things, but that makes sense considering how low their chances of success would be.

Due to the nature of democracy, rather than create an imbalance in two party vote share, one would expect this incentive structure to shift the policy positions of both parties to the Left over time. As disenfranchisement grows, the Left can move its policies further Left, and as progressive policies accumulate which the Right cannot eliminate when in power, the Right will be forced to live with policy platforms which decreasingly resemble their core ideological values.

One might also argue that the Leftist voters themselves would defend against any pathological policies, since they would be hurt by them, and would punish politicians who propose or implement them. However, I don’t think this is viable in reality. Voters, broadly speaking, aren’t capable of understanding the impacts of policies, past or present, and so cannot judge or punish politicians accordingly, except for glaring mistakes. Voters go by their personal needs and beliefs at the time, and if personal difficulties make you want progressivism more (perhaps correctly at the time, on the margin), the Left can manufacture more personal difficulties.

Another factor preventing progressive voters from defending themselves against harmful policies is dependency. It is hard to punish a party for something you don’t like if you depend on them for other things you need. This means the Left is incentivized to create dependencies in voters where it can, and not remove them when it can. With simple-to-change social issues (e.g. abortion, gay marriage, Civil Rights Legislation), both parties can lock in a segment of the populace in by siding with their position. With high effort changes like complex social issues (e.g. actual minority life outcomes, not just legal rights) and economic issues, the Left is able to give benefits to people in such a way that they will be punished for defecting, while the Right has to actually solve these problems for people to increase their vote share. 

A real world example of this would be rent control. Rent control is a great way for progressives to win votes in the short term and then lock them in. In the long term, it decreases development and building maintenance and raises prices elsewhere, which creates more disenfranchisement. This disenfranchisement, if this theory holds true, creates more progressive voters. An alternative policy, like that of maximal development so house prices are minimised, would be fantastic for individuals and the community, but this success would also probably make the community more conservative. So the Left won't pursue it because of its incentives, and the Right can't pursue it because of the difficulty of implementation. In this case, the Right's constituency would also probably include many homeowners/appartment owners who don't want their own house values to fall, even if it's what's best collectively. 

If true, this paints a bleak picture of both progressive and conservative politics within democracy. The Left increases in power when the citizens are losing, and decreases in power when the citizens are winning. Fixing issues by introducing good policy is not possible, as the Left is rewarded for bad policy, and any good policy introduced by the Right would be easy to antagonize. At best, the Right can act as a brake, but it can never acquire enough real power to fight the entropy. These incentives are adequate to ensure neither party can be good. 

I think this also explains where “grey tribers” are coming from. I think grey tribers are deeply frustrated that neither political party is seriously engaged in implementing the extropic, generative public policy options they know are available - policy options which are apolitical and truly aimed at growing the wealth and health of the nation. Thus, when performing an honest examination of the existing parties, the only reasonable conclusion they can draw is that both approaches are failing. 

This theory depends on the relationship between support for Leftism and disenfranchisement being causal. The associations with marriage, children and income are known, but it is unclear if they are causal. For marriage and children, the common interpretation I have read is that there is a causal relationship. For income the relationship is less clear. There is a correlation, but income increases with age, and so does Rightwardness, and I imagine there would be many other confounders. 

Figuring out the strength of these effects, and for what variables, seems like it would be valuable. It would show the extent to which the Left and Right are politically incentivized to create certain social outcomes. The quantitative research required for this is beyond me, but I think it would produce very interesting results were someone able to do it. 

Based on the correlations I discussed in the beginning, one would expect elites to be very Right wing. However, the opposite is true. The voting-income correlations show that the poor are predominantly progressive, with Rightism increasing with income (taking over at roughly the median income) and the trend reversing at very high incomes, with the economic elites being Leftist. Cultural elites are also overwhelmingly Leftist. My theory for this is that: if economic elites tried to individually fight the entropic forces in the system, they would lose, so they go along with them instead and use them to their advantage. Laws around banking deregulation, free trade (leading to global manufacturing), pharma laws controlling generics manufacturing and bankruptcy proof university loans have all been rigged in such a way that the system becomes more entropic as a whole, but confers benefits to a small subset of elites. For cultural elites I think the story is simpler. I think they are largely dependent on the existing political structures for funding and status, and so if these structures are always moving Left, they are incentivised to follow. 

Tldr; The Left may be incentivized to implement bad policies which produce bad outcomes, because bad outcomes generate more disenfranchisement among voters, and this may increase the progressive vote share. This is easy for the Left to do – just enable entropy. This is hard for the Right to stop or reverse – they can never acquire enough power in a democratic system of government to halt the entropy or implement extropic policy. The actual strength of these incentives is unclear to me and would require research to illuminate. If true, this would mean it is unlikely that many of our problems can be solved by introducing good policy into the current system, as neither political side is able to (the Right) or incentivised to (the Left) pull it off. 


Preferences from (real and hypothetical) psychology papers

6 октября, 2021 - 12:06
Published on October 6, 2021 9:06 AM GMT

Research projects

I'm planning to start two research projects on model splintering/reward generalisation and learning the preferences of irrational agents.

Within those projects, I'm aiming to work on subprojects that are:

  1. Posed in terms that are familiar to conventional ML;
  2. interesting to solve from the conventional ML perspective;
  3. and whose solutions can be extended to the big issues in AI safety.

The point is not just to solve the sub-problems, but to solve them in ways that generalise or point to a general solution.

The aim is to iterate and improve fast on these ideas before implementing them. Because of that, these posts should be considered dynamic and prone to be re-edited, potentially often. Suggestions and modifications of the design are valuable and may get included in the top post.

Learning textual values from textual descriptions

Parent project: this is a subproject of the value learning project.


This idea grew out of a conversation with Matija Franklin and Rebecca Gorman.

My Occam's razor paper with Sören Mindermann demonstrates that human behaviour does not provide enough data to define human preferences and values. I've grounded human values, ultimately, in our mental models, including our assessments about our own irrationality/preferences and the irrationality/preferences of others.

But it is very difficult for an algorithm to use this information. In order for an algorithm to use people's judgements to assess the irrationality or preferences of another person, the algorithm would have to know that some human action was going on, that another human was reacting to it, how they were interpreting it, that they were judging it to be irrational/rational/preference-relevant/etc... and that this judgement was to be taken as normative or as a label. So, not only is the person thinking "how irrational", but that interpretation is to be taken to be correct.


There is a shortcut to part of this process, though: psychology papers. Let's take the anchoring bias. As I've noted, we could see the anchoring bias as a preference to name numbers close-to-numbers-we-have-recently-heard. But no-one thinks about it that way.

If we look at a seminal paper on anchoring bias, it details how students were presented with products, asked if they would buy those products for the last two digits of their social security numbers, and were then asked to state the actual amount they would pay for them (the experiment). The responses were recorded (the results) and the authors commented on how they demonstrated an (irrational) "anchoring and adjustment" heuristic. This commentary might be in the abstract, introduction, or conclusion sections (or spread across other sections).

What if a fine-tuned GPT-3 or GPT-n was able to generate the author's comments from just the experiment and results? In that case the GPT-3 would be able to generate textual assessments of irrationality or bias, from textual descriptions of experiments or situations:

The blue and green arrows are ungrounded; the algorithm is merely learning the red connection. But that solves a large chunk of the problem. This algorithm would be able to infer textual descriptions of biases and preferences, from textual descriptions of situations.

Grounding the descriptions of biases and preferences (the green arrow) shouldn't be too hard: we just need to point out that words such as "values", "biases", "preferences", "heuristics" and so on, mean "values", "biases", "preferences", "heuristics" and so on.

Grounding the other side (the blue arrow) is more tricky - going from a situation to a textual description of it. This move is especially difficult if we need to add hypothetical situations as well. But it is still essentially empirical, asking how a human would describe a situation, and, as such, it can be trained on human descriptions of situations in general.

So it seems that not only does human text encode a lot of preference-relevant information, but that we can access this information, in part at least, without too much difficulty. I've been mainly thinking in terms of learning biases, but the general idea might allow us to unlock a lot of implicit human knowledge, hiding in our texts.

Research aims
  1. See if psychology research papers can be easily separated into "descriptions of experiments or situations" and "human interpretations of the experiments or situations".
  2. Generate textual assessments of irrationality or preferences from textual descriptions of situations or experiments.
  3. Check the reliability of the textual assessments, and whether they can be used as labels for assessing preferences.
  4. See what other types of human judgement information is coded, ungrounded, in texts of various types.


2021 Darwin Game - River

6 октября, 2021 - 10:29
Published on October 6, 2021 7:29 AM GMT

The River contains lots of every foragable food. The River is like the Human Garbage Dump except:

  1. It has 5x as many resources.
  2. Both air-breathing and water-breathing species can survive in the River.
  3. Organisms may not start in the River. They must wander in.

Name Carrion Leaves Grass Seeds Detritus Coconuts Algae Lichen River 500 500 500 500 500 500 500 500 Generation 0

At first, there are no animals and the foragables accumulate.

Generations 1-100

Species Original Biome Venom? Weapons Armor Speed Forage? Creator Leafy Luncheoner Temperate Forest Neither 0 0 1 Leaves Sean Hyer Mooshroom Grassland Neither 0 0 5 Grass;Seeds Vanessa slow vulture Desert Antivenom only 0 10 10 Carrion eat (Allows survival in the Desert),Corm mediocre 1-2-s Grassland Neither 1 0 2 Seeds Corm Very Lesser Forest Dragon Rainforest Neither 2 0 2 Persephone 0461

The same pattern repeats for each foragable. First the foragable accomulates. Then a species which can consume the foragable appears. Its population explodes until it has eaten the accumulated foragable. Finally, its population reverts to carrying capacity.

Persephone 0461's Very Lesser Forest Dragon. It thrived until another predator appeared which hunted it to extinction.

Generations 100-9000

Species appear. Species Disappear. Life is always in flux.

This is where all the Ocean's Soonbegons were coming from. Until they didn't anymore.

Winners Name Native Biome Venom? Attack Armor Speed Forage? Temperature Adaptation? Creator Social Media Booyahs Desert Neither 2 0 10 Carrion Heat (Allows survival in the Desert) CK None Snark Temperate Forest Neither 0 0 10 Leaves Vanessa None Venomoth Rainforest Venom + Antivenom 0 0 3 Grass aphyer None Locust Eater Eater Temperate Forest Neither 2 0 2 Seeds Multicore None Hopsplitter Grassland Neither 0 0 10 Grass;Seeds Nem None


2021 Darwin Game - Human Garbage Dump

6 октября, 2021 - 09:31
Published on October 6, 2021 6:31 AM GMT

The human garbage dump is a place you can find anything. It was also (to me) surprisingly popular with 90 species submitted.

Name Carrion Leaves Grass Seeds Detritus Coconuts Algae Lichen Human Garbage Dump 100 100 100 100 100 100 100 100

In each of our previous biomes, there was one primary source of forage. This is our first biome with many different viable options to forage. However, individually, none of them are very large. Even if there's no predators, the diverse primary production should ensure a little more biodiversity among foragers.

Generations 0-1

We start with Anti Predator Flak. It's just like just like the Ocean's First Round Predator Distractor except it was created by Multicore.

Goes Extinct in Generation Species 1 Anti Predator Flak Generations 1-10

Most of the initial populations crash, as usual.

The Seed Beetle is exactly what it sounds like: a tiny organism with the ability to digest seeds and nothing else.

Goes Extinct in Generation Species 6 Goo 6 Beauconut 8 Gelatinous Cube 8 Ivem 10 Chittering Harvester 10 Boot Nipper 10 Dump Leaf Blight Generations 10-100

The Seed Beetle Population explodes. I think what's going on is something Multicore pointed out at the game launch. The Seed Beetle is so small that it provides inadequate nutrition to large carnivores. At first, predators eat the Seed Beetle, then the carnivores starve. The Seed Beetle is thus left without predators.

The Mutant Two-Headed Speed Snail is a Grass forager with Speed 2. The Beauprey forages for Seeds and Lichen and has Speed 1.

Goes Extinct in Generation Species 11 Trash Lobster 11 Carrion Condor 11 Roaches 11 Trash Panda2 11 Muk1 12 Small CocoCrab 12 The Fart Brigade 12 Thread-Nibbler 12 Dump Aphid 12 Slime Mold 12 mediocre 2-2-s2 12 Muk2 13 Olchi 13 TrashMonster3-2 14 Sentient Bio Reactor 14 BottomFeeder 14 SpeedDemon 14 mediocre 2-2-g 14 TrashMonster2-2 15 TrashMonster3-4 15 TrashMonster4-3 16 Big CocoCrab 16 Unpalatable Coconut Crab 16 TrashMonster4-2 17 Rabbit 17 Lazy Stationary Glutton 17 sc511 17 trashSeeds 18 Grease Monkey 19 Garbage Dump Shai-Hulud 19 Trolley 20 Coconut Crabs (tiny) 20 TrashMonster3-3 21 Robot 21 Rat 21 CP445957 22 mediocre 2-2-s1 23 Frogger 23 Coconut Cruncher 23 Yonge_Dump 23 TrashMonster4-4 24 All-eating Leviathan 24 Inquisition 25 Bleh 25 Robber Crab 27 Hyper fly 28 Coconut crab 28 Slow Moving Nuclear Waste 29 Garbage Disposal 32 Blitz Leech 32 2-8-0 A algae-carrion-coconut-detritus-lichen 32 Jibbers Crabst 33 Snail-Eating Snail 34 Micro Swamp Dragon 37 Mecha giraffe 37 Twangoola 38 25kg Racing Snake 38 Redneck 40 weaponized human 44 Garbovorous Mirages 45 Dirge Bat 47 AI Lawnmower 48 Trash Locust+ 50 gumbler 50 Cubic Carrion Crawlers 56 Trash Dino 56 CarrionSpeedTank 75 COCONUT511 89 Lary 89 Gigantic Cockroach 99 Roomba Generations 100-200

Species die out more slowly but there are no dramatic shifts in population.

Goes Extinct in Generation Species 105 hoover 105 Living Garbage Can 107 Ironshell Crab 109 Zigzagoon 157 Adventurous Giraffe Generations 200-500

Nothing to see here.

Goes Extinct in Generation Species 330 Timblewinks Generations 500-1000

Nothing to…wait what?

Let's take a closer look at what's going on.

There are nine species of consequence.

Name Venom? Weapons Armor Speed Forage? Creator Seed Beetle Neither 0 0 0 Seeds DaemonicSigil Mutant Two-Headed Speed Snail Neither 0 0 2 Grass DaemonicSigil Nuclear Waste Antivenom only 0 10 0 Carrion Henny Brown Bear Neither 3 0 6 elspood Beauprey Neither 0 0 1 Seeds;Lichen Luke Trash Slime Antivenom only 0 10 0 Detritus Yull-Rete Forest Finch Neither 0 0 3 Leaves MA Snark Neither 0 0 10 Leaves Vanessa Hopsplitter Neither 0 0 10 Grass;Seeds Nem

We have four invincible foragers: Nuclear Waste consumes Carrion. Trash Slime consumes Detritus. Snarks consume Leaves. Hopsplitters consume Grass and Seeds.

The Forest Finch competes for leaves with the Snark.

The Hopsplitters compete for seeds with the Seed Beetles and the Beaupreys. The Seed Beetle is a tiny organism optimized to consume seeds. The Beauprey is less efficeint at eating seeds than the Seed Beetle but makes up for it by also being able to digest Lichen.

The Mutant Two-Headed Speed Snail eats grass, where it competes with the Hopsplitter.

The Brown Bear is a pure predator. It cannot eat the invincible foragers but it can eat the Seed Beetles, Snails, Hopsplitters, Forest Finghes and Forest Finches.

Putting this all together, we have two kinds of foragers: small foragers that can be eaten by bears and invincible foragers those that cannot. If the bear population increases the small forager population decreases which increases the invincible forager population which decreases the bear population. This cycle works in reverse too. We have a stable ecosystem.

But if it's stable then what happened around generation 690?

The Seed Beetle population decreased because the Beauprey population increased. The Beauprey Population increased because there was lots of Lichen available. The Beauprey is the only species which can eat Lichen. Around generations 550 to 650 there were very few Beaupreys. The Detritus accumulated. The Beauprey's growth around population 680 ate that accumulated Detritus. After the Detritus was consumed, the Beauprey population decreased to something stable.

Generations 1000-9000

The dynamic equilibrium is maintained until generation 8101 when the Beaupreys suddenly go extinct.

Goes Extinct in Generation Species 8101 Beauprey

Let's zoom in.

What happened is the Brown Bear population randomly rose for long enough to extinguish the Beaupreys, which were already in a precarious position due to competition from the Seed Beetles..

But that's not all. I just counted the common species. There are a few rare species hiding on the bottom edge of our graph, like Snarks. The Snark is a 10-speed Leaf eater by Vanessa. The Beauprey, having Speed 1, comes back from time to time too.

Winners Species Creator Social Media Seed Beetle DaemonicSigil Twitter Mutant Two-Headed Snail DaemonicSigil Twitter Nuclear Waste Henny None Forest Finch MA None Brown Bear elspood None Trash Slim Yull-Rete None Snark Vanessa None


Metaculus is seeking Analytical Storytellers to write essays fortified with testable predictions

6 октября, 2021 - 09:06
Published on October 6, 2021 4:44 AM GMT

About Metaculus

At Metaculus, our aim is to improve human decision-making and coordination at scale by increasing analytic capacity, reasoning, and judgment. With the unique capabilities of the Metaculus forecasting platform, our innovative programs, and our long-term partnerships, we enable a range of contributors to engage in a process of collective reasoning, forecasting, and ultimately, more informed action.

About This Role

We are hiring writers, on an ongoing basis, to produce “fortified essays” on critical topics like emerging science and technology, global health, biosecurity, economics and econometrics, technology policy, global risk, environmental science, and geopolitics. These are longform, educational essays fortified with testable predictions. 

Writers can come from many walks of life; they might be researchers, rationality enthusiasts, postdocs, data scientists, or quantitative analysts, but they must be able to write on their areas of expertise with creativity, with clarity, and with embedded, quantified forecasts. These pieces should serve as primers on crucial topics, should showcase original analysis, and should spotlight and contextualize high-leverage questions.

For examples of essays that have already been published and to get a sense of what the expectations are, see this comparison of competing theories of inflation by economist Arnold Kling, or this exploration of the future of nanotechnology by physical chemist Kevin Ausman, or this look at the spread of the Islamic State in Nigeria by geopolitical analyst Adam Ragozzino, or this piece on the past and future of the Riemann hypothesis by Ege Erdil. 

Importantly, Essays must be accompanied by original forecast questions, which analytical storytellers will need to write and themselves publicly predict on. 

Writers may submit up to five essays per month, each 1000+ words in length with at least two linked forecast questions. Essays are compensated at $300 each and at $25 per forecast question, with additional compensation awarded for especially high-quality essays attracting a significant readership. 

To Apply

Email christian at metaculus dot com with a resume or CV, a short pitch of a couple sentences for the proposed essay, and a 100+ word forecast question (ideally related to the pitch though this is not a requirement) that is formatted like and that meets the standards set by other questions on the site. Feel free to share links to or attach writing samples.


We're Redwood Research, we do applied alignment research, AMA

6 октября, 2021 - 08:53
Published on October 6, 2021 5:51 AM GMT

Redwood Research is a longtermist organization working on AI alignment based in Berkeley, California. We're going to do an AMA this week; we'll answer questions mostly on Wednesday and Thursday this week (6th and 7th of October). Buck Shlegeris, Bill Zito, myself, and perhaps other people will be answering questions.

Here's an edited excerpt from this doc that describes our basic setup, plan, and goals.

Redwood Research is a longtermist research lab focusing on applied AI alignment. We’re led by Nate Thomas (CEO), Buck Shlegeris (CTO), and Bill Zito (COO/software engineer); our board is Nate, Paul Christiano and Holden Karnofsky. We currently have ten people on staff.

Our goal is to grow into a lab that does lots of alignment work that we think is particularly valuable and wouldn’t have happened elsewhere.

Our current approach to alignment research:

  • We’re generally focused on prosaic alignment approaches.
  • We expect to mostly produce value by doing applied alignment research. I think of applied alignment research as research that takes ideas for how to align systems, such as amplification or transparency, and then tries to figure out how to make them work out in practice. I expect that this kind of practical research will be a big part of making alignment succeed. See this post for a bit more about how I think about the distinction between theoretical and applied alignment work.
  • We are interested in thinking about our research from an explicit perspective of wanting to align superhuman systems.
    • When choosing between projects, we’ll be thinking about questions like “to what extent is this class of techniques fundamentally limited? Is this class of techniques likely to be a useful tool to have in our toolkit when we’re trying to align highly capable systems, or is it a dead end?”
    • I expect us to be quite interested in doing research of the form “fix alignment problems in current models” because it seems generally healthy to engage with concrete problems, but we’ll want to carefully think through exactly which problems along these lines are worth working on and which techniques we want to improve by solving them.

We're hiring for research, engineering, and an office operations manager.

You can see our website here. Other things we've written that might be interesting:

We're up for answering questions about anything people are interested in, including

  • What needs to happen in order for the field of x-risk-motivated AI alignment research to employ a thousand ML researchers and engineers
  • The role of applied alignment research in the overall AI alignment space
  • What types of research are most promising to do
  • How alignment research might look different five or ten years from now
  • Anything related to working at Redwood

We're looking forward to answering your questions!


2021 Darwin Game - Benthic

6 октября, 2021 - 07:39
Published on October 6, 2021 4:39 AM GMT

Welcome to the depths of the sea, far beyond the reach of sunlight. Here ye shall feast upon the dead.

Name Carrion Leaves Grass Seeds Detritus Coconuts Algae Lichen Benthic 10 0 0 0 1000 0 0 0

72 species were submitted to the Benthic.

Generations 0-25

We start with a population crash, as usual. After that, the populations almost look like they stabilize. Less than half of our species have gone extinct.

Goes Extinct in Generation Species 6 Salpophore 6 Tasty 7 Marine Snow Worm 7 benthic bottom feeder 8 Scavenger 9 Isopoid 9 Detritus Eater Eater 10 Scavenger Shrimp 10 Rubble on Double 10 Sea Zombie 11 Fangliphore 11 Grumpy Jellyfish 11 Doomed 11 Happy Hunter 11 A new hope 12 Krill1 12 Sp0r3 12 Cnidophore 12 Cephalophore 12 Not Picky 13 Detritus Eater Eater Eater 14 Sea Snail 14 Hermit Crab 14 Space Horse 14 BeauOmni3 14 aBa-CG751 15 Flounder 16 Microbet 16 Slowmo 17 aBa-D273 18 Hund 18 Glyptoderm 18 Squid Kid 19 Anglerfish 19 Toxic Squid 20 Torpedo Jelly 23 Willi 24 cg-riverfish

Here are our populations at generation 25.

Population Species 170 Barnacle 158 Deepwater Devourer 124 The Infinite Depths 114 FOERDI 1,6,4,3,D 98 Flutz 88 DetriusSpeedTank 66 Spanish Beard 55 BeauOmni1 50 Silli 46 FOERDI 1,0,10,1,D 45 aBa-D262 43 Tilli 40 FOERDI 0,0,0,10,D 39 FOERDI 1,3,7,3,D 35 Pilli 34 Blood Shark 33 FOERDI 1,10,0,1,D 32 Xilli 30 Chilli 29 omastar 27 Elephantfish 27 Abyssal eel 24 Billi 19 Kraken2 17 Space Fiddler 15 Giant Squid 10 The Tideless Depths 5 FOERDI 1,9,1,1,D 5 Soonbegon 2 gonnabiteit 2 Deap Sea Tortoise 2 Flesh-Eating Clam1 1 Killi 1 Tachyphore

As you might guess, the Barnicle is an invincible Detritus eater and the Deepwater Devourer is a carnivore. However, the Deepwater Devourer is not an apex predator. It has only 3 Attack and 6 speed. (It also has Antivenom and the ability to consume Detritus.)

Generations 25-100

Barnicles dominate.

Goes Extinct in Generation Species 6 Salpophore 6 Tasty 7 Marine Snow Worm 7 benthic bottom feeder 8 Scavenger 9 Isopoid 9 Detritus Eater Eater 10 Scavenger Shrimp 10 Rubble on Double 10 Sea Zombie 11 Fangliphore 11 Grumpy Jellyfish 11 Doomed 11 Happy Hunter 11 A new hope 12 Krill1 12 Sp0r3 12 Cnidophore 12 Cephalophore 12 Not Picky 13 Detritus Eater Eater Eater 14 Sea Snail 14 Hermit Crab 14 Space Horse 14 BeauOmni3 14 aBa-CG751 15 Flounder 16 Microbet 16 Slowmo 17 aBa-D273 18 Hund 18 Glyptoderm 18 Squid Kid 19 Anglerfish 19 Toxic Squid 20 Torpedo Jelly 23 Willi 24 cg-riverfish 28 gonnabiteit 29 Killi 33 Tachyphore 35 Giant Squid 40 Space Fiddler 40 Xilli 40 Pilli 40 FOERDI 1,9,1,1,D 43 Flesh-Eating Clam1 44 Soonbegon 47 Deap Sea Tortoise 63 omastar 63 aBa-D262 65 FOERDI 1,3,7,3,D 66 Kraken2 68 BeauOmni1 73 Elephantfish 75 Billi 78 Flutz 82 FOERDI 1,6,4,3,D 83 Blood Shark 83 Silli 89 The Tideless Depths Generations 100-1000

Most of our species die out leaving only Barnicles, Spanish Beards and Tillis.

You can't see it in the above graph but a few Soonbegons trickle in to compete for Detritus.

The Barnicle and Spanish Beard are identicle invincible detritivores. The Tilli occupies a slightly different niche by foraging for Carrion too.

Goes Extinct in Generation Species 125 FOERDI 1,10,0,1,D 152 DetriusSpeedTank 170 FOERDI 0,0,0,10,D 183 Abyssal eel 203 FOERDI 1,0,10,1,D 230 The Infinite Depths 232 Deepwater Devourer 257 Chilli Winners Name Venom? Attack Armor Speed Forage? Creator Social Media Barnacle Antivenom only 0 10 0 Detritus Guy Srinivasan Spanish Beard Antivenom only 0 10 0 Detritus VJ None Tilli Antivenom only 0 10 0 Carrion;Detritus Milli None Soonbegon Venom + Antivenom 0 0 10 Detritus Martin Randall LinkedIn


2021 Darwin Game - Ocean

6 октября, 2021 - 04:59
Published on October 6, 2021 1:59 AM GMT

The Ocean has lots of algae, a little carrion and a little detritus. It was the most popular biome, with 131 native species.

Name Carrion Leaves Grass Seeds Detritus Coconuts Algae Lichen Ocean 10 0 0 0 10 0 10,000 0

To thrive in this game as a forager you must survive the initial predators. One way to do this is to breed faster than they can eat you. Another way is to use armor or another defense.

Random guy in NY took a third route. He created the First Round Predator Distractor, a cheap species with almost zero nutrition and exactly zero chance of survival that kept the predators in check for a single round.

Goes Extinct in Generation Species 1 First Round Predator Distractor

Now that Random guy in NY's silliness is out of the way let's get on to the real show.

Generations 1-50

At first the defenseless foragers thrive. Increasingly large carnivores's populations grow and crash.

Name Venom + Antivenom? Attack Armor Speed Forager? Creator Phytoplankton Neither 0 0 0 Algae aphyer Algae Tribble Neither 0 0 0 Algae simon CookieMaximizer Neither 1 0 1 Saran gkdemaree-9-eater-of-drifting-forager Neither 1 0 1 Grant Demaree Pike Neither 2 0 3 Yair Herring Neither 2 0 2 Measure Bhuphin Neither 0 2 0 Algae phenx Snail-Eating Snail Antivenom only 5 0 2 Carrion;Seeds;Detritus Taleuntum Jilli Neither 3 0 1 Algae Milli Right whale Neither 5 0 5 EricF gkdemaree-10-small-shark Neither 6 0 6 Grant Demaree gkdemaree-1-fast-shark Antivenom only 10 0 10 Grant Demaree gkdemaree-6-large-omnivore-fish Antivenom only 7 0 7 Algae Grant Demaree

Goes Extinct in Generation Species 7 Cheerless Tidehunters 12 Krabs 12 Qoxeadian 13 Bloade 13 Squidheads 14 Sleeping Sealions 16 Sanguine Scavenger 17 Terror Whale 17 Sea Snake 17 Whale 17 Algae Sprinter 17 FOERDI 1,0,10,0,CDA 18 Blue whale 18 Chybcik 18 op511 18 Crab 19 Kraken1 19 The Aparatus 19 ocean vulture 20 magikarp 20 sea tortoise 20 1-4-1 carrion 21 osc511 22 Manta Ray 22 Grerft 24 Plankton2 24 Poisonous Sponge 24 cg-fish 25 Speedy Algivore 26 cg-whale 28 RiverShark 28 Yellow Krill 28 0-0-0 algae-seed 29 Armored Algivore 30 Salt Water Crocodiles 30 speedFish 30 FOERDI 1,1,0,1,A 31 ploplo 31 Shellder 32 gkdemaree-9-eater-of-drifting-forager 33 CookiePaladin 35 CookieMaximizer 36 VenomousRiverShark 37 Little fish 37 Coconutwhale 37 CookieAshimo 38 AlgaeProbably 38 Kun 38 gkdemaree-3-defensive-venum-forager 38 gkdemaree-4-tiny-slow-fish 38 Leuphonim 38 Chasnee 39 Phytoplankton 39 Algae Eater 39 Shrimp 39 Green Krill 39 Angy Shrimp 39 Meta-Algae 39 Greenglow Bindylow 39 Smallums 39 Zooplankton 39 Anchovy 39 Guppy 39 0-0-0 algae 39 Algae Tribble 40 Plankton1 40 Microblu 40 Orange Krill 40 Krill3 40 Yonge_Ocean 40 CookieRunner 40 Blue Tang 40 FOERDI 0,0,0,0,A 41 Pike 41 Krill2 41 Fornicacious Krill 41 Red Krill 41 Scumsipper 41 Herring 41 SpeedyFeedy 41 OceanEaty 41 Beaufish 41 Khisesant 41 Senuboon 41 Bhuphin 42 River Llama 42 gkdemaree-5-small-omnivore-fish 42 Snail-tooth Grabble 42 CookieWhite 42 CookieEater 42 CookieSzybcik 42 ocean omnivore 42 cg-fleshfish 42 Jilli 42 Sea snek run 42 FOERDI 0,3,0,3,A 43 RiverSharkFood 43 Sapphire Jelly 43 fooooley 43 defenseFish 43 Rosemary 43 Bissurus 44 Right whale 44 CookieShroom 44 Armor Fish 44 Groundlings 45 Slorp 45 gkdemaree-10-small-shark 45 Emerald Jelly 45 Patricians 46 Plage 46 gkdemaree-8-slow-armored-forager 46 Nano-krill 47 Electric eel 47 CookieDevourer 47 CookieMedoKinematic 47 Buneh Rizak-e Chenar 48 Megaladon 48 gkdemaree-6-large-omnivore-fish 48 Twinkle Star 49 algae_eater 49 Lilli Generations 51-100

By generation 51, only 8 species remain.

Name Venom + Antivenom? Attack Armor Speed Forager? Creator Killer whale Neither 9 1 10 Yair gkdemaree-1-fast-shark Antivenom only 10 0 10 Grant Demaree gkdemaree-2-fast-ocean-venum Venom + Antivenom 0 0 10 Algae Grant Demaree gkdemaree-7-fast-forager-fish Neither 0 0 10 Algae Grant Demaree Apex Predator Antivenom only 10 0 10 JSH Killerspeed Antivenom only 10 0 7 Carrion VJ Flesh-Eating Clam2 Antivenom only 0 10 0 Carrion Yull-Rete Super-Armor Fish Antivenom only 0 10 0 Carrion;Algae Random guy in NY

By Round 100, all that is left are the invincible foragers with maxed-out defense. However, this time there are two niches: one Flesh-Eating Clam 2, which specializes in carrion, and another for Super-Armor Fish, which eats algae too.

Generations 101-9000

Alas, the Super-Armor Fish's ability to digest Algae lets it eventually outcompete the Flesh-Eating Clam2. A Soonbegon wanders in from time to time from to eat the accumulated Detritus.


Technically the Soonbegon doesn't survive forever (it goes extinct sometime before generation 10,000) but it survived long enough and interestingly enough I'm going to bend the rules and give it credit anyway.

Name Venom + Antivenom? Attack Armor Speed Forager? Creator Social Media Super-Armor Fish Antivenom only 0 10 0 Carrion;Algae Random guy in NY Facebook Soonbegon Venom + Antivenom 0 0 10 Detritus Randall LinkedIn


Appropriately Gray Products

6 октября, 2021 - 03:56
Published on October 6, 2021 12:56 AM GMT

(Cross posted on my personal blog.)

Epistemic status:

  • I've been in the startup world for a while, have read almost all of Paul Graham's essays, have read The Lean Startup cover-to-cover, etc. However, it's possible that I am misunderstanding/misrepresenting the idea of an MVP here. If so, I apologize and would like to be corrected.
  • As for my opinions, I feel solid about them. Not too, too confident though. To justify that amount of confidence, I would want to have had stress tested my ideas by having other people critique them. That one of the goals I have in writing this, actually.
  • This is a "starting the conversation" type of post. Not an "I've spent a ton of time researching, discussing with others, and iterating, and now I am presenting to you a finished product" type of post.

Here is how I understand the idea of a minimum viable product (MVP).

Starting out, you have some hypothesis that you want to test. Eg. that people will want to buy pet food online. How would you go about testing this hypothesis?

You could go to town and spend 18 months building out a beautiful website that has pixel perfect designs with an accompanying iOS and Android app, but that is a little bit excessive. To test your hypothesis, doing all of that isn't really necessary.

On the other hand, you could just throw together a Google Doc with products, pictures and prices and tell people to email you if they want to place an order. But I'd argue that this wouldn't be enough. You wouldn't get a good enough test of your hypothesis. People who otherwise would be willing to buy pet food online might not place orders because a Google Doc like this feels sketchy. In other words, the risk of a false negative is too high.

With that context, let me explain what an MVP says. An MVP says to keep "moving to the right" until you have a hypothesis test that is "viable", and then once you hit that point, stop. You don't want to be to the left of the line because that would mean you can't trust the results of your experiment. But you also don't want to be to the right of the line because that would mean acting on assumptions that haven't been validated.

Shades of gray

Something has always rubbed me the wrong way about this. I don't like the idea that there is this line where you declare that something is viable. "To the left of this line things aren't viable, to the right of this line things are viable." You don't hear it said this explicitly, but given how people talk about MVPs, I think that it is often times implied.

Here is how I see it. You never really get to 100% viability. There is always some chance of false negatives. Which means, the more time you spend improving the product, the smaller this chance of false negatives is. So then, you have to weigh the pros and the cons of investing more time into the product. The pro of investing more time is that you lower the chance of false negatives, and the con is that you expend resources (time, money, etc.).

In other words, viability is not black and white. It has shades of gray. So then, as an alternative to thinking about MVPs, maybe it'd make sense instead to thing about AGPs: Appropriately Gray Products.

All models are wrong, some are useful

You've heard that expression before right? That all models are wrong, and some are useful? It's something that has always stuck with me.

The reason I bring it up is because I anticipate a counterargument:

Yes, it is wrong to say that viability is black and white. You are correct in saying that it has shades of gray. However, modeling it as being black and white is useful. Who cares if it is correct.

It's a good counterargumnent. Here are my thoughts.


You know what else is useful? Having lots of different models of how the world works. This really hit me the other day as I was reading Philosophy of Therapy.

I like collecting lenses through which to view the world. Each is like a different kind of mental map that I can use to navigate the territory of reality, and just like different types of maps (some simplistic and cartoonish, others realistic and highly detailed) can be more or less useful for different purposes, even maps that I know are not literally correct can still have value.


Still, I think if more people were aware of the different lenses through which therapy can operate, they would better be able to navigate the sorts of problems that might lead them to a therapy office, maybe even help them find their way without going to one.

I'm not necessarily making the point (yet) that AGPs are a better way of looking at things than MVPs. Right now I'm just saying that they are a different lens, and that having different lenses in your toolkit is helpful. Sure, maybe one of your tools is your favorite — it's the first one you reach for and you use it more often than not — but sometimes it isn't the right tool for the job and it helps to have alternatives.

For this point about different lenses, I feel like the bar is pretty high if you want to say "No, screw all of your other lenses, you should just use this one lens."

MVP is too far to the left

I haven't spent too much time investigating alternatives to MVPs, but I know that they are out there. At one company I worked at, we used what's called a Minimum Lovable Product. MLPs are a thing. If you google it, you'll see lots of blog posts written about MLPs. The way I think about it, MLPs are like MVPs, but the line is further to the right. (Yes, you can quibble that the axis of viability is different from the axis of lovability.)

My impression is that if you look at the alternatives to MVPs, they're mostly moving the line to the right, not to the left. And I think that this trend is important. I think it points towards the line for MVPs being too far to the left.

Another thing that makes me think this, other than instincts/intuition, is reversed stupidity. I suspect that people have a decently strong bias for saying, "Ugh, this way of doing things is so dumb. I hate it so much. We should push in the other direction." And then proceed to push too hard in the other direction.

Let me be more conrete. Before MVPs, a "hail mary" or waterfall-y approach was popular, where you would spend a lot of time and money building an awesome and very feature-full product. I get the impression that people who pushed for MVPs saw this old way of doing things as stupid and reacted against it. "No, no, no! Why would you spend all of this time and money building this elaborate product without releasing it and testing it against real users!"

I agree that the old way was stupid (ie. way too far to the right), so I understand where the MVP people were coming from. I'm just saying that it is human nature to "reverse stupidity" and push too hard in the other direction. Given this human nature, I actually think the burden of proof is moreso to show that they didn't push too hard in the other direction in this specific situation. I don't see reason to believe that.


2021 Darwin Game - Desert

6 октября, 2021 - 03:24
Published on October 6, 2021 12:24 AM GMT

Our Desert is an inhospitable environment. Carrion, while nutritious, is hard to digest and there's not a lot of it. Species must be adapted to the heat too.

Carrion Leaves Grass Seeds Detritus Coconuts Algae Lichen 100 0 1 1 1 0 0 0

The most dangerous part of the desert is the predators. Two people submitted sandworms.

Name Venom Weapons Antivenom Armor Speed Creator Desert Carnivore 10 ✓ 0 10 Antihaas dp511 ✓ 10 ✓ 0 10 horu Sand Drake 6 0 10 Yull-Rete Sandworm 10 7 8 Vanessa Shai-Hulud[1] 10 ✓ 0 7 simon

Only 36 organisms were submitted to the Desert. 23 of them could digest Carrion. Carrion is very nutritious so all 23 were viable foragers.

Generations 1 to 10

A few apex predators did well.

Goes Extinct in Generation Species 6 Lesser Desert Rat 7 Cordova 8 Hyena 8 Desert-Ants 8 Desert-Foxes 8 Desert-Lizard 9 Yonge_Hot 10 Josep Raich Generations 11 to 50

The apex predators did so well they wiped out all the foragers without maxed out speed or defense. Having extinguished the easy prey, the apex predators proceeded to starve.

Goes Extinct in Generation Species 11 Desert-Coyotes 11 Cockatrice 11 cg-wildcat 12 Venomous Snark 12 Desert-Snakes 14 Bitey-the-scary 14 Smok 16 Titus 19 Desertio 20 Qanon 23 Wurm 26 Sandworm 33 Shai-Hulud 35 lDha-C293 38 Sand Drake 38 slow vulture 41 Desert Carnivore 45 Chocolattos 45 Dongles 46 dp511

Generations 51+ Name Venom Weapons Antivenom Armor Speed Eats Carrion? Other Creator Desert Viper ✓ 0 ✓ 0 10 ✓ aphyer dsc511 0 ✓ 10 0 ✓ Eats Leaves, Grass, Seeds, Detritus, Coconuts, Algae, Lichen horu Booyahs 2 0 10 ✓ CK Rock Beetle 0 ✓ 10 0 ✓ Taleuntum Desert Tortoise1 0 ✓ 10 0 ✓ Multicore Desert Tortoise2 0 ✓ 10 0 ✓ Yull-Rete Armadillo v2 0 ✓ 10 0 ✓ Henny Sol Invictus 0 ✓ 10 0 ✓ simon

Most of these speccies are identical. They're carrion foragers with maxed-out defense (including antivenom) and nothing else. Everyone who isn't a maximally-efficient invincible forager dies by generation 61.

Goes Extinct in Generation Species 60 Booyahs 61 Desert Viper

It's a random walk among equals from here.

Winners Species Player Social Media Armadillo v2 Henny None Desert Tortoise1 Multicore None Desert Tortoise2 Yull-Rete None Rock Beetle Taleuntum None Sol Invictus simon None Eratta

Simon notes that I used the wrong starting energy value of 50,000 instead of 1,000. Basically the same thing happens (except faster) when I use the correct energy value.

  1. "Shai-Hulud" is what the Fremen call sandworms. ↩︎


How can one identify the absolute safest car in practice rather than theory?

6 октября, 2021 - 00:30
Published on October 5, 2021 9:19 PM GMT

I've been considering getting a new, safer car and I am wondering how to best utilize the data available. Organizations like the IIHS and NHTSA publish information on crash test performance and safety features, and have lists of cars which should theoretically be the safest. There's also data on driver death rates, fatal accidents, and crashes by make and model.

Assuming the data is accurate, what would be a good way of approaching it to find the safest car in practice? My thinking is that the lists of safest cars should be a fairly reliable way to quickly find top contenders, and then the ones that are the safest should have the lowest driver deaths per miles driven (available in the IIHS Status Reports).

An obvious problem with this is that there's no way to factor in the driver's capability, road conditions, and other important external factors. There's also no data on new models (as there hasn't been enough time to collect it), so I would be estimating the rates of those based on the previous years' models. Another issue is that many have zero deaths recorded, which makes for lower resolution comparisons.

What else should I consider, and what other important factors am I not thinking of if my end goal is finding cars that are really a cut above the rest in terms of safety?


How should an interviewer evaluate management skills in a candidate?

5 октября, 2021 - 21:25
Published on October 5, 2021 6:25 PM GMT

My small Data Science team is hiring a new manager. I have some skill at evaluating the technical abilities of a candidate, but very little in evaluating their management abilities.

In Data Science technical interviews I don’t think one needs a picture of what domain a person works in or what the size of their team was etc. to evaluate their technical and mathematical abilities and their intelligence. Is the same true of a manager? Is so, what are the underlying dimensions of management ability that I should be trying to uncover and evaluate?