Вы здесь
Новости LessWrong.com
Examples of Measures
I recently started learning measure theory but had a pretty hard time finding realworld examples of the different measures. So this is a list of realworld examples of some different, common measures.
The count measureIf I have 4 apples, their count measure is 4. The count measure is just normal counting.
The Dirac measureYou are in a forest with some trees. It's daytime. The trees cast shadows, but the tree cover is sparse enough that there's still some sunlight that hits the ground. The dirac measure for whether you're standing in a shadow is something that outputs 1 when you're standing in a shadow, and 0 if you're not.
Imagine moving the point labeled 'you' around inside the blue region. Whenever it's inside a green circle, the dirac measure is 1; whenever not inside a green circle, it's 0.Formally, if δx.mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')} means "in a shadow when standing in position x", and A is the collection of shadow regions, the mathematical expression of this dirac measure is δx(A)={0,x∉A1,x∈A.
The dirac measure always looks like this, whatever meanings you attach to δx and the set A. If you've heard of an "indicator function", this is nearly the same thing.
(Since x is a point, the forest analogy breaks down a little: you are not allowed to be standing halfway in and out of shadow. You're in or you're out. We could fix this above by talking about "where the bajillionth atom on your left pinky toe is" instead of "where you're standing").
The Lebesque measureThe length of a line. The area of a flat shape. The volume of a 3D object. The equivalent concept in 4D, 5D, and so on. The name can be intimidating, but as a person in the physical world, you already have an intuitive idea of how the Lebesque measure should work.
An important feature of the Lebesque measure μL is that if you move your set around, μL doesn't change. An orange has the same volume whether it's in my left hand, my right hand, or balanced on your head across the room from me. Also known as "translation invariance".
A probability measure μp is a measure that satisfies the usual requirements on a probability distribution: it's restricted to the range (0, 1), its sum over that range is 1, and it satisfies a condition called "countable additivity", which you probably don't have to think about too much.
Pictured is an example with μp(1)=μp(2)=14;μp(3)=12.
Wikipedia Angle measureA measure μ∘ that sums to 360°. Pictured, μ∘(a)≈50∘;μ∘(b)≈130°;μ∘(c)=180∘.
WikipediaAngle measures are rotation invariant: rotate the picture above, and a, b, and c will still be the same angles. I think they're translation invariant, too? Seems like if you scoot that picture around, nothing changes, as long as you moved everything all at the same time.
There are more measures, but I don't understand them yet!
Thanks to the Youtube channel The Bright Side Of Mathematics for getting me from knowing nothing to knowing something, when all the other expositions I found were too hard to understand.
Discuss
Early Thoughts on Ontology/Grounding Problems
These all seem to be pointing to different aspects of the same problem.
 Crossontology goal translation: given a utility function over a latent variable in one model, find an equivalent utility function over latent variables in another model with a different ontology. One subquestion here is how the first model’s input data channels and action variables correspond to the other model’s input data channels and action variables  after all, the two may not be “in” the same universe at all, or they may represent entirely separate agents in the same universe who may or may not know of each other's existence.
 Correspondence theorems: quantum mechanics should reduce to classical mechanics in places where classical worked well, special relativity should reduce to Galilean relativity in places where Galilean worked well, etc. As we move to new models with new ontologies, when and how should the structure of the old models be reproduced?
 The indexing problem: I have some system containing three key variables A, B, and C. I hire someone to study these variables, and after considerable effort they report that X is 2.438. Apparently they are using different naming conventions! What is this variable X? Is it A? B? C? Something else entirely? Where does their X fit in my model?
 How do different people ever manage to point to the same thing with the same word in the first place? Clearly the word “tree” is not a data structure representing the concept of a tree; it’s just a pointer. What’s the data structure? What’s its type signature? Similarly, when I point to a particular tree, what’s the data structure for the concept of that particular tree? How does the “pointer” aspect of these data structures work?
 When two people are using different words for the same thing, how do they figure that out? What about the same word for different things?
 I see a photograph of a distinctive building, and wonder “Where is this?”. I have some data  i.e. I see the distinctive building  but I don’t know where in the world the data came from, so I don’t know where in my worldmodel to perform an update. Presumably I need to start building a little sidemodel of “wherever this picture was taken”, and then patch that sidemodel into my main world model once I figure out “where it goes”.
 Distributed models and learning: a bunch of different agents study different (but partially overlapping) subsystems of a system  e.g. biologists study different subsystems of a bacteria. Sometimes the agents end up using different names or even entirely different ontologies  e.g. some parts of a biological cell require thinking about spatial diffusion, while some just require overall chemical concentrations. How do we combine submodels from different agents, different ontologies and different data? How can we write algorithms which learn large model structures via stitching together small structures each learned independently from different subsystems/data?
Abstraction plays a role in these, but it’s not the whole story. It tells us how highlevel concepts relate to lowlevel, and why very different cognitive architectures would lead to surprisingly similar abstractions (e.g. neural nets learning similar concepts to humans). If we can ground two sets of highlevel abstractions in the same low level world, then abstraction can help us map from one highlevel to the lowlevel to the other highlevel. But if two neural networks are trained on different data, and possibly even different kinds of data (like infrared vs visual spectrum photos), then we need a pretty detailed outside model of the shared lowlevel world in order to map between them.
Humans do not seem to need a shared lowlevel world model in order to pass concepts around from human to human. Things should ultimately be groundable in abstraction from the low level, but it seems like we shouldn’t need a detailed lowlevel model in order to translate between ontologies.
In some sense, this looks like Ye Olde Symbol Grounding Problem. I do not know of any existing work on that subject which would be useful for something like “given a utility function over a latent variable in one model, find an equivalent utility function over latent variables in another model”, but if anybody knows of anything promising then let me know.
Not Just Easy ModeAfter poking at these problems a bit, they usually seem to have an “easy version” in which we fix a particular Cartesian boundary.
In the utility function translation problem, it’s much easier if we declare that both models use the same Cartesian boundary  i.e. same input/output channels. Then it’s just a matter of looking for functional isomorphism between latent variable distributions.
For correspondence theorems, it’s much easier if we declare that all models are predicting exactly the same data, or predict the same observable distribution. Again, the problem roughly reduces to functional isomorphism.
Similarly with distributed models/learning: if a bunch of agents build their own models of the same data, then there are obvious (if sometimes hacky) ways to stitch them together. But what happens when they’re looking at different data on different variables, and one agent’s inferred latent variable may be another agent’s observable?
The point here is that I don’t just want to solve these on easy mode, although I do think some insights into the Cartesian version of the problem might help in the more general version.
Once we open the door to models with different Cartesian boundaries in the same underlying world, things get a lot messier. To translate a variable from model A into the space of model B, we need to “locate” model B’s boundary in model A, or locate model A’s boundary in model B, or locate both in some outside model. That’s the really interesting part of the problem: how do we tell when two separate agents are pointing to the same thing? And how does this whole "pointing" thing work to begin with?
MotivationI’ve been poking around the edges of this problem for about a month, with things like correspondence theorems and seeing how some simple approaches to crossontology translation break. Something in this cluster is likely to be my next large project.
Why this problem?
From an Alignment as Translation viewpoint, this seems like exactly the right problem to make progress on alignment specifically (as opposed to embedded agency in general, or AI in general). To the extent that the “hard part” of alignment is translating from human conceptspace to some AI’s conceptspace, this problem directly tackles the bottleneck. Also closely related is the problem of an AI building a goal into a successor AI  though that’s probably somewhat easier, since the internal structure of an AI is easier to directly probe than a human brain.
Work on crossontology transport is also likely to yield key tools for agency theory more generally. I can already do some neat things with embedded world models using the tools of abstraction, but it feels like I’m missing data structures to properly represent certain pieces  in particular, data structures for the “interface” where a model touches the world (or where a selfembedded model touches itself). The indexing problem is one example of this. I think those interfacedatastructures are the main key to solving this whole cluster of problems.
Finally, this problem has a lot of potential for relativelyshortterm applications, which makes it easier to build a feedback cycle. I could imagine identifying conceptembeddings by hand or by adhoc tricks in one neural network or probabilistic model, then using ontology translation tools to transport those conceptembeddings into new networks or models. I could even imagine whole “concept libraries”, able to import preidentified concepts into newly trained models. This would give us a lot of data on how robust identified abstract concepts are in practice. We could even run stress tests, transporting concepts from model to model to model in a game of telephone, to see how well they hold up.
Anyway, that’s one potential vision. For now, I’m still figuring out the problem framing. Really, the reason I’m looking at this problem is that I keep running into it as a bottleneck to other, notobviouslysimilar problems, which makes me think that this is the limiting constraint on a broad class of problems I want to solve. So, over time I expect to notice additional possibilities which a solution would unblock.
Discuss
Signalling & Simulacra Level 3
"We lie all the time, but if everyone knows that we're lying, is a lie really a lie?"
 Moral Mazes
A common Bayesian account of communication analyzes signalling games: games in which there is hidden information, and some actions can serve to communicate that information between players. The meaning of a signal is precisely the probabilistic information one can infer from it.
I'll call this the signalling analysis of meaning. (Apparently, it's also been called Gricean communication.)
In Maybe Lying Can't Exist, Zack Davis points out that the signalling analysis has some counterintuitive features. In particular, it's not clear how to define "lying"!
Either agents have sufficiently aligned interests, in which case the agents find a signalling system (an equilibrium of the game in which symbols bear a useful relationship with hidden states, so that information is communicated) or interests are misaligned, in which case no such equilibrium can develop.
We can have partially aligned interests, in which case a partial signalling system develops (symbols carry some information, but not as much as you might want). Zack gives the example of predatory fireflies who imitate a mating signal. The mating signal still carries some information, but it now signals danger as well as a mating opportunity, making the world more difficult to navigate.
But the signalling analysis can't call the predator a liar, because the "meaning" of the signal includes the possibility of danger.
Zach concludes: Deception is an ontologically parasitic concept. It requires a preexisting notion of truthfulness. One possibility is given by Skyrms and Barrett: we consider only the subgame where sender and receiver have common goals. This gives us our standard of truth by which to judge lies.
I conclude: The suggested solution seems OK to me, but maybe we want to throw out the signalling analysis of meaning altogether. Maybe words don't just mean what they probabilistically imply. Intuitively, there is a distinction between connotation and denotation. Prefacing something with "literally" is more than just an intensifier.
The signalling analysis of meaning seems to match up rather nicely with simulacrum level 3, where the idea that words have meaning has been lost, and everyone is vibing.
Level 3 and SignallingOut of the several Simulacra definitions, my understanding mainly comes from Simulacra Levels and their Interactions. Despite the risk of writing yetanotherattempttoexplainsimulacralevels, here's a quick summary of my understanding:
 Truthtelling. An honest attempt to communicate objectlevel facts.
 Lying. For this to be meaningful and useful, there must be an equilibrium where truthtelling is common. Liars exploit that equilibrium to their advantage.
 You say X because you want to sound like the cool kids. Words have lost their inherent meanings, perhaps due to the prevalence of level 2 strategies. However, words still convey political information. Like level 1, level 3 has a sort of honesty to it: a level 3 strategy is conveying true things, but without regard for the literal content of words. We could call this level Humbug (more or less). It's also been called "signalling", but that begs the question of the present essay.
 Bullshit. Even the indirect meanings of words are corrupted, as dishonest actors say whatever is most advantageous in the moment. This level is parasitic on level 3 in the same way that level 2 is parasitic on level 1.
Here are some facts about the signalling analysis of meaning.
 There is no distinction between denotation and connotation.
 An assertion's meaning is just the probabilistic conclusions you can reach from it.
 Map/territory fit is just how well those probabilistic conclusions match reality.
 If a statement like "guns don't kill people, people kill people" lets you reliably infer the political affiliation of the speaker, then it has high map/territory fit in that sense.
 If a particular lie is common, this fact just gets rolled into the "meaning" of the utterance. If "We should get together more often" is often used as part of a polite goodbye, then it means "I want to indicate that I like you as a person and leave things open without making any commitments" (or something like that).
This sounds an awful lot like level3 thinking to me.
I'm not saying that signalling theory can only analyze levelthree phenomena! On the contrary, I still think signalling theory includes honest communication as a special case. I still think it's a theory of what information can be conveyed through communication, when incentives are not necessarily aligned. After all, signalling theory can examine cases of perfectly aligned incentives, where there's no reason to lie or manipulate.
What I don't think is that signalling theory captures everything that's going on with truthfulness and deceit.
Signalling theory now strikes me as a level 3 understanding of language. It can watch levels 1 and 2 and come to some understanding of what's going on. It can even participate. It just doesn't understand the difference between levels 1 and 2. It doesn't see that words have meanings beyond their associations.
This is the type of thinking that can't tell the difference between "a implies b" and "a, and also b"  because people almost always endorse both "a" and "b" when they say "a implies b".
This is the type of thinking where disagreement tends to be regarded as a social attack, because disagreement is associated with social attack.
This is the type of thinking where we can't ever have a phrase meaning "honestly" or "literally" or "no really, I'm not bulshitting you on this one" because if such a phrase existed then it would immediately be coopted by everyone else as a mere intensifier.
The Skyrms & Barrett ProposalWhat about the proposal that Zack Davis mentioned:
Brian Skyrms and Jeffrey A. Barrett have an explanation in light of the observation that our sender–receiver framework is a sequential game: first, the sender makes an observation (or equivalently, Nature chooses the type of sender—mate, predator, or null in the story about fireflies), then the sender chooses a signal, then the receiver chooses an action. We can separate out the propositional content of signals from their informational content by taking the propositional meaning to be defined in the subgame where the sender and receiver have a common interest—the branches of the game tree where the players are trying to communicate.
This is the sort of proposal I'm looking for. It's promising. But I don't think it's quite right.
First of all, it might be difficult to define the hypothetical scenario in which all interests are aligned, so that communication is honest. Taking an extreme example, how would we then assign meaning to statements such as "our interests are not aligned"?
More importantly, though, it still doesn't make sense of the denotation/connotation distinction. Even in cases where interests align, we can still see all sorts of probabilistic implications of language, such as Grice's maxims. If someone says "frogs can't fly" in the middle of a conversation, we assume the remark is relevant to the conversation, and form all kinds of tacit conclusions based on this. To be more concrete, here's an example conversation:
Alice: "I just don't understand why I don't see Cedrick any more."
Bob: "He's married now."
We infer from this that the marriage creates some kind of obstacle. Perhaps Cedrick is too busy to come over. Or Bob is implying that it would be inappropriate for Cedrick to frequently visit Alice, a single woman. None of this is literally said, but a cloud of conversational implicature surrounds the literal text. The signalling analysis can't distinguish this cloud from the literal meaning.
The Challenge for BayesiansZach's post (Maybe Lying Can't Exist, which I opened with) feels to me like one of the biggest challenges to classical Bayesian thinking that's appeared on LessWrong in recent months. Something like the signalling theory of meaning has underpinned discussions about language among rationalists since before the sequences.
Like logical uncertainty, I see this as a challenge in the integration of logic and probability. In some sense, the signalling theory only allows for reasoning by association rather than structured logical reasoning, because the meaning of any particular thing is just its probabilistic associations.
Worked examples in the signalling theory of meaning (such as Alice and Bob communicating about colored shapes) tend to assume that the agents have a preexisting meaningful ontology for thinking about the world ("square", "triangle" etc). Where do these crisp ontologies come from, if (under the signalling theory of meaning) symbols only have probabilistic meanings?
How can we avoid begging the question like that? Where does meaning come from? What theory of meaning can account for terms with definite definitions, strict logical relationships, and such, all alongside probabilistic implications?
To hint at my opinion, I think it relates to learning normativity.
Discuss
[Event] Ajeya's Timeline Report Reading Group #1 (Nov. 17, 6:30PM  8:00PM PT)
Ever since Ajeya's timeline report came out, I've been wanting to discuss it with other people, and see what they think about it. This is the relevant LessWrong post: https://www.lesswrong.com/.../draftreportonaitimelines
The report is split into four files, and this meeting will be about the first one. If you want to attend, please make sure that you have given the first file at the very least a very thorough skim.
If this time doesn't work well for people, I am also happy to do multiple sessions on the first report, or to reschedule it.
Structure will be mostly open discussion, with a Google Doc to help us decide on what to explore open in parallel.
The event will be happening at this link. After you spawn, walk upwards until you see the Tardis, then teleport to the library. If you get lost, just send me a message via the menu on the right and I can get you:
http://garden.lesswrong.com?code=9A40&event=ajeyasaitimelinesreportreadinggroup
Discuss
Stuart Russell at SlateStarCodex Online Meetup
Professor Stuart Russell will speak briefly on his book "Human Compatible", and then will take questions. The event begins Dec. 6, 2020 at 20:30 Israel Standard Time, 10:30 Pacific Standard time, 18:30 UTC.
Please register here and we will send you an invitation.
Stuart Russell is a Professor of Computer Science at the University of California at Berkeley, holder of the SmithZadeh Chair in Engineering, and Director of the Center for HumanCompatible AI. His book "Artificial Intelligence: A Modern Approach" (with Peter Norvig) is the standard text in AI, used in 1500 universities in 135 countries. His research covers a wide range of topics in artificial intelligence, with an emphasis on the longterm future of artificial intelligence and its relation to humanity.
Discuss
Specialized Labor and Counterfactual Compensation
Note: the math didn't import. For now you may want to read it at the original source.
I have three purposes in this post. The first is to review the formal game theory found in Robert Ellickson's Order Without Law. It's not a large part of the book, but it's the part that I'm most qualified to judge. Not that I'm a formal game theorist myself, but I'm closer to being one of them than to being any kind of social scientist, historian or lawyer. If his formal game theory is nonsense, that would suggest that I ought to discount his writing on other fields, too. (Perhaps not discount it completely, especially because formal game theory is outside his main area of study. Then again, lots of the book is outside his main area of study.)
Spoiler alert: I think he holds up reasonably well. I want to ding him a few points, but nothing too serious, and he possibly even contributes a minor original result.
My second purpose, which is valuable for the first but also valuable of itself, is to try to extend it further than Ellickson did. I don't succeed at that.
My third is simply to be able to cut it from my inprogress review of the rest of the book.
Ellickson discusses two games. One is the classic Prisoner's Dilemma, in which you either Cooperate (for personal cost but social benefit) or Defect (for personal benefit but social cost).1 The other he calls Specialized Labor, in which two people must choose whether to Work on some common project or Shirk their share of it. It differs from the Prisoner's Dilemma in two ways. First, it's asymmetrical; one player is a less effective worker than the other, and gets less payoff from Working while the other Shirks than does the other player. The other is that in this game, the socially optimal outcome is Work/Shirk, not Work/Work.
(Many authors consider that the second change isn't really a change, and that a Prisoner's Dilemma can perfectly well have Cooperate/Defect be socially optimal. So they'd say Specialized Labor is simply an asymmetrical version of the Prisoner's Dilemma. In my taxonomy I define the Prisoner's Dilemma more narrowly than that; see also this comment. Ellickson uses the same narrow definition as me. I'd instead say Specialized Labor is an asymmetrical version of Too Many Cooks.)
Note that payoffs aren't measured in utility. They're measured in something Ellickson calls "welfare". He doesn't really explore the formal consequences of this. But what it gives us is that, since welfare is supposed to be objective, we can sum different people's welfare; when I used the phrases "social cost" and "socially optimal" in the previous paragraphs, talking about the sum of both players' results, that was a meaningful thing to do. I'm not sure exactly what it costs us, except that I don't expect results about mixed strategies to hold. (Someone won't necessarily prefer "50% chance of 3 welfare" to "certain chance of 1 welfare". I wasn't planning to consider mixed games anyway.) We can still assume that people prefer higher amounts of welfare to lower amounts of it.2
I'm going to pretend that Cooperate and Defect are also called Work and Shirk, so that I don't have to use both names when talking about both games.
In normalform, these games look like this:
Prisoner's Dilemma Player 2 Work Shirk Player 1 Work \( ww_* \), $ ww_* $ $ ws_* $, $ sw_* $ Shirk $ sw_* $, $ ws_* $ $ ss_* $, $ ss_* $ $ sw_* > ww_* > ss_* > ws_* $, and $ 2ww_* > sw_* + ws_* $ Specialized LaborPlayer 2 Work Shirk Player 1 Work $ ww_* $, $ ww_* $ $ ws_1 $, $ sw_* $ Shirk $ sw_* $, $ ws_2 $ $ ss_* $, $ ss_* $ $ sw_* > ww_* > ss_* > ws_1 > ws_2 $, and $ 2ww_* < sw_* + ws_1 $
How to read these symbols: the subscript is the player who gets the payoff, the first letter is their move, and the second letter is the other player's move. If the subscript is \( * \), then this combination is symmetric.3 So \\( ws_1 \\) is the payoff to Player 1, if he Works while Player 2 Shirks. $ ws_2 $ is the payoff to Player 2, if she Works while Player 1 Shirks.4 $ ws_* $ is both of these values, when they're equal to each other. And to be clear, when they're equal, $ ws_1 $ can stand in for $ ws_* $ just as easily as the other way around.
To help make the structure more visible, I've colored the symbols in green or red according to local incentive gradients  green for "this player prefers this outcome to the outcome they get from changing their move", red for the opposite of that. So when $ ws_1 $ is red, that means $ ss_1 > ws_1 $, since $ ss_1 $ represents Player 1's payoff if he changes his move while Player 2 keeps hers the same. A quadrant is a Nash equilibrium (meaning "neither player wants to change their move unilaterally") iff it has two green symbols. I've also given a slightly darker background to the socially optimal quadrants.
Comparing these games, Ellickson claims for example that norms will tend to punish someone who Shirks in a Prisoner's Dilemma, rather than rewarding those who Work, because eventually most people will Work and it's cheaper to sanction the thing that happens rarely. But in a Specialized Labor game, norms will tend to reward the efficient worker ("cheapest laborprovider") for Working, because that encourages people to obtain the skills necessary to perform this work. There's a background level of skills that everyone is expected to have, and people are punished for falling short of them and rewarded for exceeding them.
So most of the points I want to ding Ellickson here are because this is kind of a strange choice of games. For one thing, it seems to assume that: teaming up to work is more expensive than working individually, iff players have unequal skill levels.
Honestly I don't think that's so implausible as a heuristic. I think "most work projects have gains from working together" is a decent guess, and then one way to remove those gains could be if one player is much more skilled than the other. Still, Ellickson doesn't make this argument, or acknowledge that the assumption is kind of weird.
Another way to justify the omission is if the ommitted possibilities don't add much of interest. Prisoner's Dilemma and Specialized Labor are opposite corners in a twobytwo grid parameterized by "synergistic/discordant" (gains or no gains from cooperation) and "symmetrical/asymmetrical". If our tools for working with them can also be applied to the other corners without much extra effort, then there's no need to consider the others in detail. More on this later.
Something weird on the face of it is that in Specialized Labor, Work/Work results in the same payoff to both players. Why assume that that's symmetrical? But I don't think this is a big deal. Plausibly people can calibrate how hard they work if they think they're getting a worse result than the other. Also I suspect you just don't change much by allowing it to be asymmetrical, provided that both payoffs are in between $ sw_* $ and $ ss_* $.
Similarly you might suppose that the efficient worker doesn't just pay less to Work than the inefficient worker, he also does a better job. In which case we might want to set $ sw_1 < sw_2 $. But again, I doubt that matters much.
Here's my largest objection: Ellickson doesn't consider that work might be worth doing selfishly. In both games, you maximize your own outcome by Shirking, and if that means the work doesn't get done, so be it. But that puts a narrow band on the value of a piece of work. From a social perspective, it's not worth doing for the benefits it gives to one person, but it is worth doing for the benefits it gives to two. I think a lot of the situations Ellickson looks at don't really fit that model. For example, building a fence seems like something you'd often do of your own accord, simply for the benefits it gives to yourself, but Ellickson considers it a Prisoner's Dilemma because most people have the relevant skills. (He doesn't analyse whether fencebuilding is more easily done in tandem.)
To model this possibility, we'd set $ ws_1 > ss_* $, and maybe $ ws_2 > ss_* $ as well. This gives the game that I like to call the Farmer's Dilemma and others call Chicken, Hawk/Dove or Snowdrift. (Here's why I call it that.) Normally I think of the Farmer's Dilemma as symmetrical, but the asymmetrical case seems fine to count as an instance of it, at least right now.
The tricky thing about this game is that even though you'd be willing to do the work yourself if no one else benefitted, the fact that someone else does benefit makes you want them to join in and help with the work. If they decline, your only ingame way to punish them is not to do the work, which hurts you too  but if you don't punish them, you're a sucker. This is fundamentally different from the tricky thing with Prisoner's Dilemma and Specialized Labor, which in both cases is simply that people have no selfish incentive to work. So it seems like an important omission. Especially because depending on the exact payoffs, it may be that "one player is a sucker while the other makes out like a bandit" is both a Nash equilibrium and socially optimal.
The thesis of the book is to propose a certain hypothesis. Roughly speaking, and for the purpose of this essay, we can assume the hypothesis says: norms will evolve to maximize the aggregate welfare of the players.
(And so Farmer's Dilemmas might be a good place to look for failures of the hypothesis. When the socially optimal result is for one player to be a sucker, and that's also a Nash equilibrium, the hypothesis thinks this is fine. Humans might not think that, and norms might evolve that the hypothesis would have ruled out. But note that this is only the case in the Discordant Farmer's Dilemma  when there are no gains from cooperation. In the Synergistic Farmer's Dilemma, the socially optimal result is for both players to Work. The Discordant Farmer's Dilemma might be rare in practice  I wouldn't expect it with fencebuilding, for example.)
Let's pretend we're creating a system of norms for these games. Something we can do is mandate transfers of welfare between players. In each quadrant, we can take some of one player's payoff and give it to the other. Total payoff stays the same, and so the socially optimal outcome stays in the same place. But the distribution of welfare changes, and the Nash equilibria might move.
How do we encourage the socially optimal result by doing this? This is Ellickson's possible minor contribution. He points out that we can do it by introducing a debt from those who Shirk to those who Work, and that the value $ ww_*  ws_1 $ works in both these games.
He calls this the "liquidatedKantian formula" but doesn't explain the name, and I have only a vague understanding of where he might be going with it. Since the name hasn't caught on, I'm going to propose my own: counterfactual compensation. If I Shirk, I compensate you for your losses compared to the world where I worked.
(To compare: actual compensation would be compensating you for the losses you actually suffered from working, $ ss_*  ws_1 $. Actual restitution would be handing over to you the gains I got from your work, $ sw_*  ss_* $. Counterfactual restitution would be handing over to you the gains I got from not working myself, $ sw_*  ww_* $. Each of these takes one player's payoff in one quadrant, and subtracts the same player's payoff in an adjacent quadrant. Compensation is about your costs, and restitution is about my gains. The actual variants are about differences between the world where no one worked and the worlds where one of us worked; they're about the effects of work that actually happened. The counterfactual variants are about the differences between the worlds where only one of us worked and the world where we both worked; they're about the effects of work that didn't happen.)
(Also: yes, obviously there are caveats to apply when bringing this formula to the real world. Ellickson discusses them briefly. I'm going to ignore them.)
If we apply this formula to the Prisoner's Dilemmma, we get this:
Prisoner's Dilemma with counterfactual compensation Player 2 Work Shirk Player 1 Work $ ww_* $, $ ww_* $ $ ww_* $, $ sw_* + ws_*  ww_* $ Shirk $ sw_* + ws_*  ww_* $, $ ww_* $ $ ss_* $, $ ss_* $ $ sw_* > ww_* > ss_* > ws_* $, and $ 2ww_* > sw_* + ws_* $Since $ ww_* > sw_* + ws_*  ww_* $, this puts the incentives in the correct place. The Nash equilibrium is now for both players to Work, which is socially optimal.
(In my taxonomy, depending on whether $ sw_* + ws_*  ww_* ≷ ss_* $, this new game is at the point where The Abundant Commons meets either Cake Eating or Studying For a Test. It's not unique in either case, because there are at most three distinct payout values.)
Specialized Labor is more complicated. There are three ways we might decide to apply counterfactual compensation. We could say that the Shirker compensates the Worker for the Worker's costs, either $ ww_*  ws_1 $ or $ ww_*  ws_2 $ depending on who Worked. Or we could say that the Shirker compensates the Worker for what the efficient Worker's costs would have been, $ ww_*  ws_1 $ regardless of who Worked. Or we could say that the efficient worker never owes anything to the inefficient worker; he gets to just say "sorry, I'm not going to pay you for work I could have done more easily". Lets call these approaches "actualcosts", "efficientcosts" and "substandarduncompensated"
Ellickson doesn't discuss these options, and I ding him another point for that. He just takes the substandarduncompensated one. Here's what it looks like.
Specialized Labor with counterfactual compensation (substandarduncompensated) Player 2 Work Shirk Player 1 Work $ ww_* $, $ ww_* $ $ ww_* $, $ sw_* + ws_1  ww_* $ Shirk $ sw_* $, $ ws_2 $ $ ss_* $, $ ss_* $ $ sw_* > ww_* > ss_* > ws_1 > ws_2 $, and $ 2ww_* < sw_* + ws_1 $Player 2 has no incentive to Work, regardless of what Player 1 does, because $ ss_* > ws_2 $ and (unlike in the Prisoner's Dilemma) $ sw_* + ws_1  ww_* > ww_* $. And given that Player 2 is Shirking, Player 1 has incentive to Work. So again, we've moved the Nash equilibrium to the socially optimal quadrant.
This isn't, like, a mindshattering result that's going to blow open the field of game theory. But I don't remember seeing it before, and Ellickson doesn't attribute it to anyone else. I'm inclined to give him some credit for it. Even if others have had the insight before  which I expect they have  it seems like he's still doing competent work in a field outside his own. Not amazing work, not particularly difficult work, but competent.
One objection: the inefficient worker gets a better result than the efficient worker. That seems bad to me, because it discourages people from becoming the efficient worker. I don't think this is a big deal, though. For one thing, acquiring skills probably does increase your own payoff; your skills will feed into $ ww_* $, not just your $ ws $. (So it directly increases your payoff in Work/Work, and reduces your debt in Shirk/Work.) Someone else acquiring skills will increase your payoff even more, perhaps, but that's not a big problem. For another thing, such incentives can be handled outofgame. I do think Ellickson should have acknowledged this issue, and I ding him a point for not doing so. But a brief note would have been fine.
What happens if we apply counterfactual compensation in the other possible ways? The only difference is in the bottom left quadrant, which becomes either $ sw_* + ws_2  ww_* $, $ ww_* $ (actualcosts) or $ sw_* + ws_1  ww_* $, $ ww_* + ws_2  ws_1 $ (efficientcosts). The problem with both of these is that that quadrant might now be a Nash equilibrium. In the first case, Player 1 might prefer that quadrant over Work/Work, depending on $ 2ww_* ≷ sw_* + ws_2 $, and Player 2 will certainly prefer it over Shirk/Shirk. In the second case, Player 1 will certainly prefer that quadrant over Work/Work, and Player 2 might prefer it over Shirk/Shirk, depending on $ ww_* + ws_2  ws_1 ≷ ss_* $. That's not great, we only want a Nash equilibrium in the socially optimal quadrant.
On the other hand, I note that if $ ws_1  ws_2 $ is small, then the social cost is low; and if it's large, then (except perhaps with some fairly specific payoff values?) that quadrant isn't a Nash equilibrium. Meanwhile, if payoffs are uncertain  if people might disagree about who the more efficient worker is  then either of the other choices seems more robust. And this is more of an aesthetic judgment, but it feels like the kind of aesthetic judgment that sometimes hints at deeper problems: there's something a bit weird about how substandarduncompensated is discontinuous. A small change in Player 2's skills lead to a small change in her compensation in each quadrant, until she gets equally skilled as Player 1, at which point there's a large change in the Shirk/Work quadrant.
On the other other hand, a feature of how these games translate to the real world is that players encourage each other to discuss in advance. Someone building unilaterally may not get to claim this debt. So if they disagree about who the efficient worker is, that's unlikely to cause much grief.
What about measures other than counterfactual compensation? Actual compensation ($ ss_*  ws_1 $) doesn't work. If a player expects the other to Shirk, they'd be indifferent to Working; and in a Prisoner's Dilemma, if they expect the other to Work, they might prefer to Work or not depending on $ ww_* ≷ sw_* + ws_1  ss_* $. (In Specialized Labor, that inequality always resolves as $ < $ which gives the incentives we want.)
Actual restitution ($ sw_*  ss_* $) is sometimes okay in a Prisoner's Dilemma, but if $ ws_* + sw_* < 2ss_* $ then Shirk/Shirk remains a Nash equilibrium; players will only want to Work if they expect the other to also Work. In Specialized Labor it has the problem that players would prefer to Work than to pay restitution, and so Work/Shirk cannot be a Nash equilibrium.
Counterfactual restitution ($ sw_*  ww_* $) has much the same problem in a Prisoner's Dilemma; if $ ws_* + sw_* < ww_* + ss_* $ then Shirk/Shirk is a Nash equilibrium. And in both games, a player who expects the other to Work will be indifferent to Working.
There are other options for payment one might consider; I haven't even looked at all of them of the form "one raw payoff minus another raw payoff". But so far, counterfactual compensation seems like the best option.
(We could even consider values of the debt based on information outside of the original payoff matrix. But Ellickson points out that when deciding how to act in the first place, players will already want to figure out what the payoff matrix looks like. If the debt was based on other information, there'd be a further cost to gather that information.)
While we're here, let's look at the other games implied by Prisoner's Dilemma and Specialized Labor. The Asymmetrical Prisoner's Dilemma (or Synergistic Specialized Labor) has $ ws_1 ≠ ws_2 $ but $ 2ww_* > ws_1 + sw_* $. In this case, counterfactual compensation does exactly what we want it to do, just like in the symmetrical Prisoner's Dilemma; except that substandarduncompensated is no good, it doesn't give us a Nash equilibrium at all. (Player 1 prefers Shirk/Work to Work/Work, and Work/Shirk to Shirk/Shirk. Player 2 prefers Work/Work to Work/Shirk, and Shirk/Shirk to Shirk/Work.) If Ellickson had considered this game, he'd have had to discuss the possible ways one might apply counterfactual compensation, which would have been good. So I ding him a point for it.
Symmetrical Specialized Labor (or Discordant Prisoner's Dilemma, or Too Many Cooks) has $ ws_1 = ws_2 $ but $ 2ww_* < ws_* + sw_* $. The difficulty here is that there's no way to break the symmetry. Any of the three ways to apply counterfactual compensation will be equivalent, and leave us with two Nash equilibria in the two socially equal quadrants. The "discuss in advance" feature saves us again, I think; players don't need to somehow acausally cooperate to select one to Work and one to Shirk, they can just, like, talk about it. So I think it was basically fine for Ellickson to not consider this game, though it would have been worth a brief note.
How does this work in the Farmer's Dilemma? First we need to clarify exactly what set of games that refers to. In symmetrical games, I think of it as having $ sw_* > ww_* > ws_* > ss_* $; that is, each player would prefer the other to do all the work, or failing that to help; but they'd still rather do it all themselves than for the work not to get done.
I'm going to break symmetry by separating $ ws_1 $ from $ ws_2 $ as before. Without loss of generality, we can specify $ ws_1 > ws_2 $, but I'm not going to decide whether $ ws_2 ≷ ss_* $. It might be that only one player is skilled enough to benefit from Working alone.
So in normal form, the Farmer's Dilemma looks like this:
Farmer's Dilemma Player 2 Work Shirk Player 1 Work $ ww_* $, $ ww_* $ $ ws_1 $, $ sw_* $ Shirk $ sw_* $, $ ws_2 $ $ ss_* $, $ ss_* $ $ sw_* > ww_* > ws_1 > ss_*$, and $ ws_1 > ws_2 $Either of the top two quadrants could be socially optimal, depending whether the game is synergistic or discordant (that is, whether $ 2ww_* ≷ sw_* + ws_* $). Shirk/Work may or may not be a Nash equilibrium, depending whether $ ws_2 ≷ ss_* $. So how does it look with counterfactual compensation? I'll consider the synergy and discord cases separately.
Synergistic Farmer's Dilemma with counterfactual compensation (substandarduncompensated) Player 2 Work Shirk Player 1 Work $ ww_* $, $ ww_* $ $ ww_* $, $ sw_* + ws_1  ww_* $ Shirk $ sw_* $, $ ws_2 $ $ ss_* $, $ ss_* $ $ sw_* > ww_* > ws_1 > ss_*$, $ ws_1 > ws_2 $, and $ 2ww_* > sw_* + ws_1 $Oh dear. Substandarduncompensated compensation is clearly not going to work; Shirk/Work might still be a Nash equilibrium. In Specialized Labor it was fine that the efficient Worker would prefer the inefficient Worker to do all the work, because the inefficient worker would say "nuts to that". In a Farmer's Dilemma she might continue to Work, which we don't want. Even if we specified $ ws_2 < ss_* $, we'd simply have no Nash equilibrium; like in the Asymmetrical Prisoner's Dilemma, one player would always get a better result by changing their move.
Fortunately, either of the others seems fine. The payoffs for these are the same as in Specialized Labor, but their values have changed relative to adjacent quadrants. Actualcosts gives us $ sw_* + ws_2 − ww_* $, $ ww_* $ in that quadrant, which isn't a Nash equilibrium because $ ww_* > sw_* + ws_2 − ww_* $. (Compared to this quadrant, Player 1 would rather Work and Player 2 would rather Shirk.) And efficientcosts again gives us $ sw_* + ws_1 − ww_* $, $ ww_* + ws_2  ws_1 $, which isn't a Nash equilibrium because $ ww_* > sw_* + ws_1 − ww_* $. (Player 1 would still rather Work. Player 2 may or may not prefer to Shirk; if $ ws_2 > ss_* $ she'll certainly prefer this quadrant, might prefer it even if not, but it's not a problem either way.)
What about the discordant case? If $ ws_2 < ss_* $ we actually already have the desired result. The only Nash equilibrium is Work/Shirk which is socially optimal. But as discussed above, it's a crap result for Player 1, and my sense is that the "no incentive to become the efficient worker" problem now becomes a lot more of an issue. Let's see what happens with counterfactual compensation.
Discordant Farmer's Dilemma with counterfactual compensation (substandarduncompensated) Player 2 Work Shirk Player 1 Work $ ww_* $, $ ww_* $ $ ww_* $, $ sw_* + ws_1  ww_* $ Shirk $ sw_* $, $ ws_2 $ $ ss_* $, $ ss_* $ $ sw_* > ww_* > ws_1 > ss_* $, $ ws_1 > ws_2 $, and $ 2ww_* < sw_* + ws_1 $Again, substandarduncompensated doesn't really help; Shirk/Work will be a Nash equilibrium iff it was one before. But at least Player 1 gets a lessbad result from Work/Shirk. (Player 2 still does better than him.)
Actualcosts might also be a Nash equilibrium in that quadrant, if $ ww_* < sw_* + ws_2 − ww_* $. And so might efficientcosts, if $ ww_* + ws_2  ws_1 > ss_* $. (Again, this always holds if $ ws_2 > ss_* $, so looking only at the Nash equilibria, this is strictly worse than having no compensation.)
So this is unfortunate. We can't reliably remove that Nash equilibrium with counterfactual compensation. Depending how we apply it, we might even make it an equilibrium when it wasn't before.
(Actual restitution also works in the synergistic game, but moves the Nash equilibrium to Work/Work in the discordant game. Counterfactual restitution makes players indifferent to Working if they expect their partner to Work, so in practice I guess Work/Work is the Nash equilibrium there, too. And actual compensation would be negative, which is silly.)
Summing up, counterfactual compensation:
 Gives people good incentives in Prisoner's Dilemma. In an Asymmetrical Prisoner's Dilemma, substandarduncompensated doesn't work.
 Gives people good incentives in Specialized Labor, using substandarduncompensated. Mostlygood incentives using the other implementations.
 Gives people good incentives in the Synergistic Farmer's Dilemma, except that substandarduncompensated only works sometimes.
 Maybe kinda sorta helps a bit in the Discordant Farmer's Dilemma. Maybe not.
So that's not amazing. I do think the Discordant Farmer's Dilemma is just fundamantally, in technical terms, a real bastard of a game. But even in the synergistic variant, the way we calibrate it to get the best incentives is different from the way we calibrate it for the best incentives in Specialized Labor.
So I appreciate Ellickson's contribution, and I think it's a real one. But it's not as much as we might have hoped. I think he had a blind spot about the Farmer's Dilemma, and his tools don't really work against it. He also would have done well to consider counterfactual compensation schemes other than substandarduncompensated.
With counterfactual compensation in mind, Ellickson proposes a variant Iterated Prisoner's Dilemma tournament, and a strategy for it that he calls "EvenUp". EvenUp takes advantage of features of the tournament that make it more realistic, and is modelled on realworld behaviours that he describes elsewhere in the book.
The tournament has rounds of both Prisoner's Dilemma and Specialized Labor, and payoffs for them can vary considerably. He suggests that perhaps one in five rounds might have each payoff increased twentyfold. Additionally, in between rounds, players can unilaterally choose to make a side payment to their partner.
To apply the EvenUp strategy, a player would use an internal balance to keep account of standing with their partner. Whenever counterfactual compensation would be owed, according to the analysis above, they'd adjust the balance by its value. (Ellickson doesn't specify, but presumably they'd also adjust whenever their partner makes a payment to them.) Whenever the balance was close to zero, they'd play the socially optimal strategy. If they were in debt, they'd make a side payment. And if they were in credit, they'd "exercise selfhelp": Shirk when they'd otherwise Work.5 (But only if the debt owed was more than half the value of the compensation, so that the balance would become closer to zero.)
There are three parameters I might be inclined to play with. One: which variant of counterfactual compensation should we use? (Ellickson's wording doesn't make it clear which he intends. Above he took substandarduncompensated for granted, but is wording here sort of hints ambiguously at efficientcosts. He doesn't note or justify the change if there is one.) As noted, substandarduncompensated gives the right incentives where the other options sometimes don't. Still, I wouldn't be surprised if the other options sometimes helped to avoid a feud (a loop of mutual defections or alternating defections).
Related, two: suppose we do use substandarduncompensated. When in credit, and facing a Specialized Labor game as the efficient worker, should we Shirk? (Since we'd never Work as the inefficient worker, this is the only time the choice of counterfactual compensation variants is relevant.) Regardless of the other player's move, no compensation is owed. So Shirking will destroy communal resources, but not bring players' standings back in balance. On the other hand, it does stop us from extending more credit that may never be paid back. It may be worth having a higher threshold for this than for Shirking in a Prisoner's Dilemma, but I'd say never Shirking in this case would be a mistake.
And three: is "brings the balance closer to zero" the optimal condition to use for when to exercise selfhelp? If we exercise it more readily, others may be more inclined to cooperate with us in the first place, but that effect is probaby minor  there's only so much we can be exploited for, over the whole game. On the other hand, we're also destroying more total payoff, per round. It may be worth only exercising selfhelp if our credit is more than say threequarters the value of counterfactual compensation.
(If we're looking at modfications to the tournament: as well as tweaking the probability distribution of the various possible payoff matrices, I'd be interested to see what changes if you add a small or large transaction fee to the side payments. Naturally I'd also like to see what happens if you add the possibility of Farmer's Dilemmas, but then EvenUp needs to be altered to account for it. Of other games in the genre, a Discordant Abundant Commons ($ ws_1 > ww_* > \{ sw_*, ss_* \} $, $ 2ww_* < ws_1 + sw_* $, and I'm not sure what the constraints on $ ws_2 $ should be) would also be a good addition. Maybe an asymmetrical AntiCoordination variant, with a single socially optimal outcome so as not to confuse SociallyOptimalOutcomeBot. The others don't seem like they'd add much; they all have $ ww_* $ as the highest payoff, so their socially optimal outcomes are also individually optimal. That doesn't mean there's no reason not to play Work, but the reasons mostly boil down to "I'm willing to hurt myself to threaten or punish you" and you already get that from the Farmer's Dilemma. So I'm not convinced the other games add much strategic depth, and they do add noise.)
Ellickson predicts that EvenUp would do well in this tournament, and I agree. It's inexploitable, rewards its partners for cooperating, forgives past transgressions, and plays well against itself. I'd be concerned about what happens if it plays against some similar strategy with different ideas of fairness  might you get into a situation where only one of them is ever satisfied at a time, leading to alternating defections? More generally I just don't trust either myself or Ellickson to have especially reliable intuitions about this.
Ellickson also says that if EvenUp turns out not to be evolutionary stable  that is, if a society of EvenUp players can be exploited by other strategies, or wouldn't be able to enter groups currently dominated by other strategies  his hypothesis would no longer be credible. I think it would be stable, but even if not, I'd be somewhat forgiving. I'd want to know why not, before deciding how it reflects on the hypothesis.

Strictly speaking: if you Defect, that always harms your opponent and benefits yourself, relative to you Cooperating. And if your opponent Cooperates, this will always be a social cost as well, harming your opponent more than it benefits you. But if your opponent is also Defecting, then the structure of a Prisoner's Dilemma is agnostic on whether your defection is a social cost; it might benefit you more than it harms your opponent. ↩

I'm not actually sure we can assume that, but that question is out of scope. ↩

This notation is kind of experimental on my part. Ellickson instead uses symbols $ A, B, C, D, E $ in descending order, but that makes it hard to remember which goes where in the grid. And when I look at the Farmer's Dilemma later on, the ordering will be lost, making it even more confusing. ↩

I tossed a coin to choose how to assign pronouns. ↩

Incidentally, in in a typical IPD tournament, with only Prisoner's Dilemmas and no variance in payoffs, EvenUp plays identically to TitforTat. An EvenUp player would never be in debt in such a tournament, since they'd never Shirk except to correct a balance, and doing so would either leave the balance unchanged (if their partner also Shirked) or bring it to exactly zero. ↩
Discuss
University of Tübingen, Master's Machine Learning
The purpose of this post is to give an overview of the Machine Learning (ML) Master’s program at the University of Tübingen. It should function as guidance for people who are interested in studying ML and weigh the pros and cons.
This article is part of a series of articles on different European master's programs related to artificial intelligence and machine learning.
General OverviewThe ML program in Tübingen has 120 credits, 30 of which are assigned for the thesis. The program has three mandatory courses, Deep Learning, Statistical Machine Learning, and Probabilistic Machine Learning. All other courses can be chosen more or less freely with some small restrictions, e.g. they have to be in the broad area of ML. The full range of lectures can be found in the module handbook, though not all of them exist yet. Since 2019 was the first year of the Master, I expect these gaps to be closed in the next two years. In the winter semester 20/21 there are already many new courses.
Regarding prerequisites: There are some specifications on the website but they can be a bit vague. According to the creator of the program, who also oversees admissions, absolutely necessary criteria are having a sufficient understanding of proofbased math (e.g. through a math or Computer Science (CS) Bachelor’s degree) and a basic understanding of algorithms and other CS concepts. To give you a prior probability for a successful application we can only look at the first iteration of the degree where aonethirdround 150 people from all around the world applied, 60 were deemed to be sufficiently qualified and around 4060 actually started the Master. The program probably could have handled more students but the creator decided that applicants need to pass a certain level of skill. This acceptance rate of one third does not seem very low but I expect it to get more competitive in the future. Last time the program was only announced three months before the application deadline and already 150 people applied. Since then the University of Tübingen/the MPI has had more exposure within the media and the official ML YouTubechannel hit 800 subscribers in its first week I would expect the program to become more competetive. An alternative in case of rejection is to apply for the CS Master’s and transfer to the ML Master’s later. However, if too many people use this loophole it might be closed or ML students might be prioritized in contested lectures.
I think the general selfunderstanding of the program is one of excellence, i.e. it wants to produce people who have a deep understanding of the current ML landscape. As far as I can tell Tübingen seems to put a lot of emphasis on the theoretical understanding of ML (all courses have practical exercises too) but it’s hard to judge without an explicit comparison. The second emphasis in Tübingen is the social component of ML. There are lots of seminars discussing the ethics of ML or the intersection between ML and other fields such as medicine. From my personal experience, I would estimate that around 75 percent of the lectures that I attended fulfill the idea of excellence, i.e. they teach a mixture between old but relevant and new material, require a lot of effort but yield great understanding. Unfortunately, I had some courses that were rather shallow, didn’t update their content even though new research was available, or were clearly too easy. Since most lectures are in their first iterations and five new professorships in the realm of ML have been filled in the last two years and the university is still hiring more I expect the average quality of the lectures to rise further.
The student population that I know so far (only the first generation, so small sample size) is roughly 50 percent German and 50 percent international but I expect them to become more international in the future. From my perspective they are on average rather highperforming, ambitious students confirming the selfunderstanding of excellence.
The core lectures of the course are Deep Learning, Probabilistic Machine Learning, and Statistical Machine Learning. If you want to peak into the lectures you can find them on YouTube. There are around 20 further ML related lectures including Mathematics of ML, Data Literacy, Time Series, SelfDriving Cars, Neural Data Analysis, and Efficient ML in Hardware  just to name a few (For a full list look at the module handbook). Additionally, you can choose between around 50 different general CS lectures to broaden your perspectives.
Since only 24 credit points of 120 are mandatory lectures the program allows for individual specializations. Currently, one can specialize in applied or theoretical ML but it is impossible to focus exclusively on, for example, Reinforcement Learning. Given the impressive amount of new MLrelated professors and groups, I expect that specialization will be easier in the future.
Regarding Corona, die University of Tübingen has adapted quite well and all lectures are now online. If you want a sample of the average quality of lectures, I would recommend looking at the YouTube channel.
The grading scheme in Tübingen is similar to other programs in Germany, i.e. it is rather hard but possible to achieve the best grade of 1.0. It is also realistic that you fail a class if you have not prepared for it and there is little grade inflation compared to e.g. the US.
If you already want to publish at conferences during your Master’s program, most supervisors will support you if you are willing to put in the effort. My supervisor, for example, told me that the aim of my Master’s thesis was to submit it to ICML if the results were sufficiently good (I started with a CS master and switched to ML if you were wondering about the timeline). However, whether you want to go through this effort and try to publish is obviously up to you and your supervisor, I can only say that most potential senior researchers would be up for it and willing to support the effort.
Research Directions in TübingenThe amount of ML research that is done in Tübingen is huge. There is Deep Learning, Probabilistic ML, Statistical ML, Computer Vision, Robotics, some Reinforcement Learning (RL), SelfDriving Cars, Robustness and Adversarial examples, some Natural Language Processing (NLP), Fairness, and Ethics in AI (from a technical and humanities perspective), ML in Climate Science, a very large Neuroscience section, Causality and much more. I think the fields that are currently a bit underrepresented are NLP, RL, and AIsafety. Some years ago, Tübingen didn’t have a large focus on Deep Learning but they have upgraded and adapted since then and I would expect them to be a global tier 2 when it comes to Deep Learning.
Some of the fields that Tuebingen is internationally known for include Probabilistic Numerics (Philipp Hennig), Empirical Inference with a focus on causality and Kernel methods (Bernhard Schölkopf), Robustness and Optimization (Matthias Hein), Selfdriving cars and Computer Vision (Andreas Geiger), and the neuroscience groups (Bethge lab, Peter Dayan). If you are interested in the intersection between ML and neuroscience I would suggest doing the ML master. If your focus lies with the foundations of neuroscience there are other master programs in Tübingen that might be a better fit.
Personally, I first did a bit of everything for a year and then specialized in the overlap of probability theory and Deep Learning by working on Bayesian Neural Networks. I think for most subfields in ML it can be said that somebody in Tübingen is working on them and if you are interested in specializing very early I would recommend clicking through the links below.
If you want to check out who does ML research in Tübingen, have a look at the research groups, people’s page of the IMPRS, the website of the MPIIS, and the ML in science cluster of excellence.
Options outside of UniversityTübingen is right at the heart of the cybervalley initiative. This essentially just means the province and local industry come together to boost the ML competence in the region. They fund new professorships, research groups, buildings, etc. In short, Tübingen spends a lot of money on ML. Being an Excellence Cluster is not just a label but comes with a 50 Million Euro grant over 7 years that started in 2019. Its aim is to attract global talent at the intersection of ML and other sciences (e.g. climate science, ML for social good or ML in medicine). The benefits of such a cluster are indirect but noticeable. Many of the cluster people offer seminars which means that you can discuss the implications of ML with domain experts (e.g. a philosopher or geologist) and gain new perspectives. Additionally, the cluster organizes workshops and small conferences that are usually free for students where you can broaden your perspectives.
If you are leaning more to the research side, you can try to become a student assistant, write your thesis or do research projects with the university or at the Max Planck Institute for Intelligent Systems (MPIIS) and thereby have direct access to top researchers in their respective field (this is not exaggerated, just look at the latest news).
If you care more about industry experience there are also lots of options. You can do internships or collaborations with large companies like Bosch or IBM. Bosch and Amazon are both building an AI campus for 700 and 200 researchers respectively that should be finished in the next couple of years. Even though their buildings are not built yet, they already do collaborations with the university.
Some personal notesEven though this sounds a lot like a promotional piece, I honestly think that Tübingen is the place to be for ML, at least in continental Europe. However, if you want to do research in the fields of NLP, RL, or AI safety other universities might be a better fit. Even though I am not sure if there are Master’s programs with a strong focus on AI safety in Europe or even globally.
Regarding the courses: I have taken most of the ones that are already available and I think the majority of lectures and seminars are good with some exceptions. To figure out which ones you should avoid, I would recommend asking more experienced students. The vast amount of different options definitely is a benefit, especially since they are likely to become even more in the future.
Tübingen as a town might not be for everyone. At the end of the day, it is not a large city but a town of 90k inhabitants (35k of which are students) that has fewer options (nightlife or food diversity) than a larger city could provide. However, the university provides a lot of options for physical activity and there are other ways to spend your free time. If you really want a “big city feeling” though, you will likely not find it in Tübingen.
From an Effective Altruism (EA) perspective, Tübingen is pretty nice. There is an active EA chapter, we are currently founding an AIsafety reading group, and there is a small LessWrong chapter. There are also many other university groups and NGOs, like a debating club or Global Marshall Plan, that might be interesting from an EA perspective.
Decision GuideYou should consider the ML Master’s program in Tübingen if you
 Want to have a 2year/120 ECTS tuitionfree master
 Want theoretical and practical courses in your program
 Want to have the option to cooperate with industry (e.g. Amazon, Bosch) or academic (e.g. MPIIS, Cluster of excellence) collaborations
 Want the option to explore the intersection between ML and other sciences (e.g. ethics in AI or ML in Medicine)
You should not choose Tübingen if you
 Want to have a big city feeling to your place of study
 Want to focus primarily on topics of Natural Language Processing or Reinforcement Learning
If you have any further questions about the town or program, want to get advice on how to improve your chances of getting in, or just want to leave some feedback don’t hesitate to contact me via the channels listed on my blog. If you want to know more about the research I do you can find short summary posts on my blog.
Discuss
University of Edinburgh, Master's Artificial Intelligence
This article is part of a series of articles on different European master's programs related to artificial intelligence and machine learning.
Basic data on the degree Duration: 1 year (or 2 years parttime)
 Cost: ~15.000€ p.a. for EU students (before Brexit)
 90 ECTS
The main focus of this text is to help you decide, whether studying an MSc AI degree at the University of Edinburgh (AI@Ed) is an option for you. It is not supposed to give detailed technical information like links to admission pages or tips for how to find a room in Edinburgh. You will need this kind of information once you decide to apply to UoE. I hope that I can guide your decisionmaking process with this article.
This is also not an advertisement, I have no affiliation with the University of Edinburgh. It's just an honest and objective opinion on the degree, the university, and the city.
Text is by: Marco Kinkel, feel free to message me on hi@marcokinkel.de
The degreeCourses and AreasBeing able to choose from a variety of topics was important to me. AI research is manifold and includes grasping a notion of intelligence from different viewpoints. I was happy to choose from interdisciplinary courses on Cognitive Science, Neuroscience, Robotics, Bioinformatics and of course many core ML courses. The university is especially known for its research and lectures on Bayesian ML, Computer Vision, Natural Language Processing and Biomedical Sciences. Here is a list of most important courses (which obviously changes over the years). The bold courses can be considered the standard of the degree, you will find the majority of your friends in these. However, depending on your interests, it can be worth thinking outside the box and taking a less popular course. It can expand your horizon and you will get better teaching because they are less crowded (e.g. 10 instead of 200 students).
 ML
 Introductory Applied Machine Learning
 Machine Learning and Pattern Recognition
 Machine Learning Practical
 Reinforcement Learning
 Vision
 Image and Vision Computing
 Language
 Accelerated Natural Language Processing
 Automatic Speech Recognition
 Natural Language Understanding, Generation, and Machine Translation
 Data Science
 Text Technologies for Data Science
 Data Mining and Exploration
 Design and HCI
 Case Studies in Design Informatics
 HumanComputerInteraction
 The Human Factor: Working with Users
 Bio and Neuroscience
 Computational Cognitive Neuroscience
 Bioinformatics
 Misc
 Probabilistic Modelling and Reasoning
 Computational Cognitive Science
 Natural Computing
 Robotics: Science and Systems
 Algorithmic Game Theory and its Applications
 Introductory Informatics Courses (for students from other fields)
 Introduction to Practical Programming with Objects
 Computer Programming for Speech and Language Processing
 Programming Skills
 Additionally, you can choose one or two courses from other schools, including courses like
 Robotics, AI and the Law
 The Computational Mind
 Ethics of Artificial Intelligence
You can find descriptions of these courses in the Degree Regulations and Programme of Study (DRPS) for the academic years 2019/2020 and 2020/2021. Unfortunately, the current Covid19 situation results in a cancellation of many courses for the academic year starting in September 2020, as you can see in this list.
As you can see, AI@Ed does not force you in a particular area of AI. Many courses exist on Machine Learning and Language Processing, but you can always choose to flavour your degree with diverse (but due to the time limit of one year slightly superficial) knowledge in AIrelated fields like robotics, neuroscience, language processing, philosophy of mind and computational psychology. The variety of courses has its drawbacks: it is difficult to choose only 6 to 8 courses within the very short time of one year. This is a general downside of the degree to which I will come back later. From an EA standpoint it is important to notice that while some courses mention AI ethics or AI sustainability, no course specifically focuses on AI alignment or safety.
In addition to the choosable courses, you will have two mandatory courses that are supposed to prepare you for the final thesis: Informatics Research Review (IRR) introduces academic writing and citation techniques and is just additional practice in academic writing. Informatics Project Proposal (IPP) on the other hand is actually useful because you collect literature and write a proposal document for the topic of your dissertation.
For people with a less technical background, there are courses to enhance your programming skills. These are useful but not necessary because most courses don't require a lot of programming experience. My friends with backgrounds in Physics, Neuroscience or Statistics easily succeeded without these additional courses. If you have used a Jupyter Notebook before and used Pandas, R or Matlab to inspect some experimental data, you’re all set. You don’t need any knowledge on subjectspecific Python libraries, as those will be introduced in the respective courses. A good introduction to MLrelated Python libraries is the labs of the lecture Introductory Applied Machine Learning.
Generally, the amount of code you will encounter depends a lot on your courses. In most courses the lectures impart theoretical knowledge which is applied in their tutorials and labs. If you’re interested in coding and practicing ML model design, implementation and parameter tuning, you should take Machine Learning Practical. Other courses like Game Theory are purely theoretical.
Influential Researchers at UoE Chris Williams (Turing Fellow; ML, statistical pattern recognition, probabilistic graphical models and computer vision)
 Mirella Lapata (computational models for the representation, extraction, and generation of semantic information)
 Amos Storkey (Bayesian and Neural Systems)
 Sharon Goldwater (unsupervised learning of linguistic structure)
 Stephen Renals (development of interactive systems that can understand human communication)
 Iain Murray (Probabilistic Machine Learning and Inference)
 Peggy Series (Computational Psychiatry)
 Simon King (Speech processing)
 Michael Gutmann (Probabilistic Modelling and Reasoning)
 Matthias Hennig (Computational Neuroscience)
You will have lectures with between 50150 other students from different Informatics MSc degrees (Informatics, AI, Data Science, Cognitive Science, ...). Most main lectures are not very individual with 150+ students, but the tutorials and labs are divided into small groups. In tutorials, you typically discuss exercises that go somewhat deeper into the lecture material. The tutors are mostly Ph.D. students who can be more or less motivated or talented for a tutoring job. In labs, you complete assignments and other exercises with the help of (often not very helpful) instructors and your fellow students. The attendance to lectures and tutorials is only mandatory for nonEU students due to visa regulations (note that this might change after Brexit).
The lectures have very different qualities. In the UK, I think, professors are not obligated to hold lectures, which increases their interest and hence the lecture quality. But in my personal experience, one of five lectures is still so bad that it may be better to just read the slides (but this is probably the case in every university).
You will encounter some 'inverted classroom' lectures, where the material is provided as videos and the actual lectures are QA sessions. There are very few courses where you do actual research (that could be published). In Machine Learning Practical where you develop and evaluate your own ML models, you have a lot of freedom, which comes closest to actual ML research.
Generally, all lectures are recorded, so you can rewatch them as often as you like. This is very useful in the coursework period (middle of semester, see below), where you will hardly have the time to attend all lectures.
Most courses contain two to four hours of lecture plus one or two hours of tutorials and labs per week. A typical week in my first semester looked like this:
You can see that it is quite full if you attend all tutorials and labs (which you should in the beginning to keep up the pace). The material is conveyed very quickly, so being sick for a week is not a rewarding experience. Between lectures, you will find yourself sitting in Appleton Tower (see below) doing assignments and catch up on lecture materials. This leads to the second point of criticism: the worklife balance is rather bad. Since every lecture has midterm assignments worth between 10 and 50% of the final mark, it's difficult to keep up with the lecture material during the semester. Hence, you will not have a lot of fun in the weeks of the exam period. I cannot compare the worklife balance in AI@Ed to other MSc degrees from my own experience but I've heard of degrees with less stressful semesters and exam preparations.
Exams and DissertationThe UK has its own marking scheme, which seems pretty selfexplanatory at first glance:
Mark (%)GradeDescription90100A1An excellent performance, satisfactory for a distinction8089A2An excellent performance, satisfactory for a distinction7079A3An excellent performance, satisfactory for a distinction6069BA very good performance5059CA good performance, satisfactory for a master's degreeEverything below that is a fail. However, a mark higher than 75% is only given for extraordinary performance, i.e. if the quality of your work is publishable. Therefore, many exams contain open questions worth 25% to keep students from getting too high marks. The general range of grades is therefore somewhere between 60 and 75 percent. This can be confusing for recruiters outside of the UK, who might think that a 73% degree is rather bad. Note that the 'excellency' range is as large (25%) as the rest of the marking range, so whether your work is deemed excellent, and if so, how excellent, is rather unpredictable. From our experience, excellency sometimes just means a huge amount of extra work, but this is again not guaranteed to give you >75%. This critique applies to all universities in the UK, not only UoE.
Usually, you will have one coursework per course within the semester. It consists of applied exercises on the lecture material, but it can go way beyond. Completing it will take a considerable amount of time, which is stressful since you have a coursework (CW) for every course. Your effort should depend on the weight of the CW, which is usually between 10% and 50% of the final course mark. However, the CW is marked by many different tutors in a more or less unmoderated process, so you can have frustratingly bad luck with your marker.
In contrast, the dissertation process is organized very well. You can propose a thesis project topic yourself or you can choose from a huge list of offered projects. The selection process is not interesting here but be assured that you will definitely find an exciting project in your field of interest, some even in collaboration with Amazon (although it's hard to get them). You will start with the dissertation project after all exams and lectures are completed (mid May) and finish after 3 months (mid August). This is a very short time for a dissertation, but most of the supervisors are aware of that and hence define the topics narrowly. I don't know about anyone who published a research article from their dissertation but this should be possible. However, bear in mind that you only have 3 months. This is a downside for people aiming for a PhD after their degree. But don’t despair,many MSc thesis supervisors will offer you a PhD position if you did well in your thesis.
Here is a public archive of outstanding dissertations from 2019, to get a glimpse of the variety.
Oneyear degreeAlthough it is compelling to do a oneyear MSc degree, to quickly proceed to the next level, this short time also has its disadvantages:
 No contact to master students in higher year
 you have to think about the thesis (selfproposed) and a Ph.D. (application deadlines) very early on
 you can take only a few courses
 you don't have a lot of free time to explore the city and country, because the dissertation is written in the holidays
I want to address the problem of specialization again: If you already know which field to specialize in, you're fine with a oneyear degree. If on the other hand, your aspiration is to get a broad interdisciplinary knowledge about AI, that's perfect too! But you cannot get both in one year. Say you spend your 68 courses on Introductory Applied Machine Learning, Computer Vision, Natural Language Processing, Computational Cognitive Science, Robotics, Computational Neuroscience, Data Mining, and Game Theory. Then you had one course in each of those fields, which is great for a broad overview but rather bad if you want to call yourself a professional afterward. I would advise against doing a oneyear degree if you still want to explore the diverse AIrelated research areas and specialize in some subfield.
University and organizationAdmissionAs for most AI programs nowadays, it's very hard to get in. First of all, you need two reference letters and good grades. According to this website, the offer rate is about 14%. This coincides with my experience of having exceptionally smart and diligent fellow students.
Buildings, working areasUoE's lecture halls are very modern and large. If you don't take unusual courses, they are all within a radius of a 5minute walk and all within the south of the city. A nice park (the Meadows) is nearby where you can spend some time in the sun (if it's out).
As a busy student, you will spend most of your spare time studying in Appleton Tower, that is a modern 9floor building dedicated to informatics students. It contains lecture halls, large lab spaces with PCs, and seminar rooms which can be used for studying in groups or individually. In exam times you can find people basically living there, which is possible thanks to the kitchen areas. When you see the modern interior equipment in the working areas, labs, and lecture halls, you know where the student fees go.
While Appleton Tower is specifically for School of Informatics students, the main library is open for everyone and offers additional workplaces. However, in the exam period, you will have difficulties finding a spot there.
Every big building like Appleton Tower, David Hume Tower, and the Library have a small cafe on the ground floor. Here you can get hot beverages, snacks, and at lunchtime even a small selection of hot food. I will address the food problem in a second.
Organisation, ITO, Student RepsOfficially, the university is very open for communication and there are many channels to approach. We have a very friendly staff at the Informatics Teaching Organisation (ITO), who is responsible for all official studenttouniversity issues (exams, lecture organization, tutorial group assignment, ...). The ITO together with the student representatives work very well for organizational issues. However, it is very difficult to get in touch with the lecturers, the researchers, and their departments. Bad decisions in the university's upper management led to a massive influx of students in the past years, which overwhelmed the staff and led to long chains of communication. The highprofile researchers are flooded with requests for supervision and have to cut their research to cope with organizational tasks. If you are really interested to work with one specific researcher, you will eventually get in contact with them but it's difficult to get a broad overview of the research at UoE because you can't just walk in the departments and have a chat. You cannot even get close to the research departments, because they are located in a building you can only enter with an appointment. This also frustrates the researchers at UoE, which is why they participate in strikes.
StrikesDuring my one year in Edinburgh, we had two strikes organized by the University and Colleges Union (UCU). The reasons for the strike are manyfold. One of them is the massive increase in student numbers (which increases the UoE turnover) together with the decreasing spending for staff and organization. You can find more reasons here.
Many lecturers and academic employees participated and even students solidarised, which led to buildings being closed, and some lectures being canceled. The demands have not been fulfilled since, so additional strikes can be expected in the next few years.
Clubs and ActivitiesA very curious component of university life in the UK are clubs. You will find a club for every thinkable hobby or interest (Harry Potter Club, Skydiving Club, Atheism and Humanism Club, Beneficial Artificial Intelligence Club, etc). Clubs always welcome new members and are a great opportunity to try new activities. However, the program is so stressful that you don't have much time for activities anyway. Most people's free time activities were restricted to going to the gym. This of course depends on how ambitious you personally are, there are rumors of people who actually have time for other extracurricular activities.
EdinburghFoodThe university, unfortunately, has no central canteen with cheap food. There is a large selection of wrap, soup, and sandwich places in the area and the cafés in the university buildings also sell some food, so you will not starve, but the lunch becomes rather expensive over time (e.g. £5 for a wrap. If you fancy a nice hummus falafel wrap I recommend the instagram account that writes indepth reviews on each of them in Edinburgh). It makes sense to bring your own food and use the microwaves in the university buildings.
AccommodationThe university dorms are usually more expensive than private housing and you can only apply to them after you received an unconditional offer (which happens rather late), so I would recommend looking for private shared flats on SpareRoom. The rent is high in Edinburgh, you can expect to pay £450 to £650 per month for a room.
EdinburghEdinburgh is a beautiful city in a beautiful country. Although it has half a million inhabitants, it feels like a small town if you avoid the tourism spots. This is easy as a student because the university area is south of the tourism center. If you live close to the uni, you will not need any public transport ever. Uni, pubs, supermarkets are all within a walking distance.
The city also has a lot to offer, with many (overpriced) attractions such as the castle, but also beautiful and free places such as Holyrood Park. The city offers lots of pubs, nice places to eat, and cultural activities. You can easily avoid the tourism areas (except in August with the Fringe Festival).
The winters in Edinburgh are cold, windy, rainy and dark, and depending on your accommodation, going inside doesn't help much. So prepare for that by bringing warm clothes and buying vitamin D supplements and a SAD lamp or light box. However, the summer is beautiful and if you have the time you can swim in the sea, go hiking or just roam around in the green parks.
SummaryAI@UoE considers itself as an 'elite' program in Europe. Considering the acceptance rate and the intelligence and diligence of the students, this is definitely true. However, it depends on your course choices and a bit of luck, whether you receive an 'elite' education. Some courses and lecturers are more challenging than others, which often leads to a better learning outcome, but a worse worklife balance. I'm not sure where the pressure comes from, but some lecturers feel the urge to compete with other elite universities when it comes to the course contents and speed ("We cannot cut topics because we must compete with a similar course at the University of Oxford''). This can be frustrating, but again, you will learn a lot more in these challenging courses.
Many highprofile researchers work at UoE and this degree can be a great starting point for a subsequent Ph.D. with one of them. However, the general direction of the university described in the strike section increasingly demotivates the staff and leads to less 1:1 communication for students, including the increasing difficulty to get in touch with those highprofile researchers. This will only become a problem if you are planning to build a network and connect with the local research staff. If you plan to just get your degree and move on, you will probably not be impacted by this issue.
AI@UoE offers a wide range of lectures with a good portion of most AIrelated topics. You can get an interdisciplinary degree (including philosophy, neurobiology, psychology, cognitive science, robotics) or focus on core ML ideas. However, you will not find many courses covering EA related topics like AI Alignment and social impact of AI. The university is very modern and provides nice spaces to study and to collaborate. Finally, Edinburgh is the perfect mixture of a large city with lots of activities and a small town where you can live and study without being distracted by tourists. I would
You should do the AI@UoE degree, if... ... you know which fields you want to specialize in
 ... you want to do a quick 1year degree
 ... you like Edinburgh and Scotland
 ... you want to gain broad knowledge about AI and a specialization in a subfield
 ... you desire 1:1 communication with researchers
 ... you want unimpeded progress without possibly being affected by strikes
 ... you are more interested in the societal impact or possible beneficial applications of AI than in technical aspects
Discuss
European Master's Programs in Machine Learning, Artificial Intelligence, and related fields
While there is no shortage of detailed information on master’s degrees, we think that there is a lack of perspectives from students that have actually completed the program and experienced the university.
Therefore we decided to write articles on multiple European master's programs on Machine Learning, Artificial Intelligence, and related fields. The texts are supposed to give prospective students an honest evaluation of the teaching, research, industry opportunities, and city life of a specific program. Since many of the authors are Effective Altruists and interested in AI safety a respective section is often included as well.
It may not always be obvious, but there are many Englishlanguage degrees across Europe. Compared to America, these can be more affordable, offer more favorable visa terms, and a higher quality of life. We hope that you will consider bringing your talents to Europe. These are the articles that have already been written:
 University of Amsterdam (Master's Artificial Intelligence)
 University of Edinburgh (Master's Artificial Intelligence)
 ETH Zürich (ML related M.Sc. programs)
 EPF Lausanne (ML related M.Sc. programs)
 University of Oxford (Master's Statistical Science)
 University of Tübingen (Master's Machine Learning)
 University of Darmstadt (Master's Computer Science with ML focus)
 University College London (coming soon)
 University of Cambridge (coming soon)
This selection of Master’s programs is not an ultimate list of “good” master's programs – we just happened to know the authors of the articles. If you want to add an article about any MLrelated program anywhere in the world don’t hesitate to contact us and we will add it to the list. We also don't claim that this is a complete overview of the respective programs and want to emphasize that this does not necessarily reflect the perception of all students within that program.
Authors: Marius (lead organizer), Leon (lead organizer), Marco, Xander, David, Lauro, Jérémy, Ago, Richard, James, Javier, Charlie, Magdalena, Kyle.
Discuss
On Arguments for God
This post is about God, but of course, it isn't really about God, but about a particular pattern in general.
We're pretty much all in agreement here that God doesn't exist.
I have no doubts that this correct, but it also poses a trap.
Suppose there are forty arguments for God. Even if we know definitely for a fact that God doesn't exist, it doesn't mean that all of these forty arguments are wrong.
It would if all of these arguments claimed to definitely prove that he doesn't exist, but not if some of these arguments only claim he is more likely to exist than not or that God is not as unlikely as we might think.
In fact, it'd actually be suspicious if all forty of these arguments came out against God. Surely we should expect the advantage to belong to the deists in at least one or two?
But since we have very good reasons to believe God doesn't exist and someone presents us with those arguments, surely we'll assume that they have to be wrong. And so we'll search very hard, until we find something that is plausibly an error or at least more plausible than God and talk ourselves into believing it.
And once we've introduced that first error, we've opened up the door for more.
Discuss
What are some good examples of fake beliefs?
I'm reading through Rationality from AI to Zombies again and just finished the section on Fake Beliefs. I understand the ideas and now I'm trying to ask myself whether the ideas are useful.
To me, the biggest reason for them being useful is if I myself have some fake beliefs, it'd be good to identify and get rid of them. I don't think I do though.
Another reason why they might be useful is to identify them in others and better understand what is going on. Actually, this doesn't seem particularly useful, but it at least is somewhat interesting/satisfying. Anyway, I'm having trouble thinking of good examples of fake beliefs in others. The examples in the blog posts seem pretty contrived and perhaps uncharitable to what is actually going on in their heads.
So, to wrap this question up, I am most interested in hearing about examples of fake beliefs that rationalists might be prone to having, but I am also interested in hearing any other examples.
Discuss
My Confusion about Moral Philosophy
Something about the academic discussion about moral philosophy always confused me, probably this is a more general point about philosophy as such. Historically people tried to arrive towards truths about objects. One used to ask questions like what is right or wrong. Then one started to ask what the definition of right and wrong could be. One could call that Platonism. There is the idea of truth and the game is won by defining the idea of truth, a chair or a human in a satisfying way. I claim the opposite is true. You can define an object or an idea and the definition of the idea makes the idea to a useful entity which one can develop further. At least this would be the right way to philosophies in my opinion. Something similar is done in mathematics too to my knowledge. Axioms seem to be the beginning. On a few axioms in math all theorems and all sentences seem to be built upon. Change the axioms or subvert them then one would end up with a totally different system of mathematics, with different theorems and sentences most likely. However the main difference in this analogy is that we know of the axioms in mathematics to be true on an intuitive level. That’s the unique difficulty of philosophy. We do not seem to have axioms in philosophy. We could however make a somewhat reasonable assumption that if one of the foundational axioms will prove wrong the system of mathematics might entangle itself in contradictions or at least in some inconsistencies. Historically this did. in fact happen. To give an historical example there was a Fundation crisis of mathematics in the second half of the 19. century and in the early 20. century. Therefore one could argue that the same could happen to philosophy once philosophy is evolved enough. Now I will explain my confusion about moral philosophy.
Moral philosophy seems to me to be a judgement about once own utility function. You can basically choose if you care more about being just to people, maximizing their utility or doing what is regarded as honorable by your peers. You can choose if you want to include animals, plants or just humans in your considerations. There does not seem to be a right answer in the sense that a right answer would have a special pair of attributes. In the usual academic discussion of utilitarianism, deontological ethics or virtue ethics there will always appear something that makes a theory problematic, therefore one will abstain from fully committing to one of the mentioned systems, of which their are of course several different versions. What confuses me a bit is that those problems will change anyone’s mind. A strict utilitarian will necessarily get in conflict with some considerations of justice. That should not surprise someone because if one started deciding to be utilitarian one defined a scope about things one will care and about which things one will not care. The true reasons one might be uncomfortable with the implications of the trolley problem is that one violates his utility function which precisely does not care about the academic discussion of it, but cares about the felling of guilt and shame. Morality is motivated by our feelings and our philosophy about it is just an attempt to make or feelings that evolutionary evolved consistent. The rational way to deal with once morality seems therefore for me to be to just make sure one minimizes guilt, shame and maximizes the pleasure that helping others will give most people. If one assumes that we can not control our moral sentiments or do away with it, we could have a inconsistent moral system without compromising our rationality. Because it’s inconsistency contributes to our moral enjoyment and minimizes our moral suffering.
At the beginning I described mathematics. And I described that it’s foundations relies on Axioms. It seems to me that one could describe a whole school of thought in philosophy on the foundation of rationality. Instead of asking in moral philosophy what is right to do? Which is determined by vague notions of right. One could ask what is rational to do? Rationality is far easier to define and inconsistencies can exist as long a consistency with the idea of rationality is present. This will of course not end the discussion about moral philosophy, but could show that it isn’t as relevant for humans to a certain extent. This mode of thinking could be extended to other fields too. For example to politics. Instead of concerning oneself in political philosophy to such an large extent on legitimation questions one could concern oneself more about what rational legislators or governments should do. Rationality of a government could even play a part in legitimizing it.
Discuss
Sharding the Brigade
The Secular Solstice is planning on using the bucket brigade singing app that some friends and I have been building. While the events we've hosted so far have been something like 20 people, this might be 500 or more. I've spent some time over the last couple weeks figuring out how to scale it, and I think it's in a good place now.
I started, as one always should, with profiling. By far the biggest amount of time was being spent in Opus encoding and decoding. This is what lets us send audio over the internet at decent quality without using an enormous amount of bandwidth, so it's not optional. We're also already doing it in C (libopus with Python bindings) so this isn't a case where we can get large speedups by moving to a lowerlevel language.
The first question was, if we pulled off encoding and decoding into other processes, could we keep everything else in the same process? The app is built around a shared buffer, that everyone is constantly reading and writing from at various offsets, and it's a lot nicer if that can stay in one process. Additionally, there is a bunch of shared state that represents things like "are we currently in a song and when did it start?" or "who's leading?" that, while potentially separable, would be easier to keep together.
I split the code into a "outer" portion that implemented decoding and encoding, and a "inner" portion that handled everything else. Running a benchmark [1], I got the good news that the inner portion was fast enough to stay all in one process, even with a very large number of clients:
$ python3 unitstress.py 2.45ms each; est 245 clients 2.44ms each; est 246 clients 2.46ms each; est 243 clients $ python3 unitstress.py inner 0.05ms each; est 11005 clients 0.05ms each; est 11283 clients 0.05ms each; est 11257 clientsSince encoding and decoding are stateful, I decided that those should run in longlived processes. Each user can always talk to just one of these processes, and it will always have the appropriate state. This means we don't have to do any locking, or any moving the state between CPUs. I don't know of a great way to manage many sharded processes like this, but since we only need about eight of them we can do it manually:
location /echo/api/01 { include uwsgi_params; uwsgi_pass 127.0.0.1:7101; } location /echo/api/02 { include uwsgi_params; uwsgi_pass 127.0.0.1:7102; } ...
This also meant creating echo01.service, echo02.service, etc. to listen on 7101, 7102, etc.
Once we have our codec processes running, we need to way for them all to interact with global state. After playing around with a bunch of ideas, I decided on each codec process (client) having a shared memory area which is also open in a singleton global process (server). The client can make blocking RPCs to the server, and because it's so fast that's not a problem that it's blocking.
I decided on a buffer layout of:
1 byte: status (whose turn) 2 bytes: json length N bytes: json 4 bytes: data length N bytes: data To make an RPC, the client fills the buffer and, as a final step, updates the status byte to tell the server to take its turn. The server is constantly looping over all of the shared memory buffers, and when it sees one that is ready for processing it decodes it, updates the global state, writes its response, and updates the status byte to tell the client the response is ready.The protocol is the same in both directions: arbitrary JSON (10kB max), then audio data (200kB max). This means that when we want to pass a new value through shared memory we don't need to update the protocol, but it does mean the server has a bit more processing to do on every request to decode / encode JSON. This is a fine tradeoff, since the only part of the system that is too slow is dealing with the audio codec.
I set up a reasonably realistic benchmark, sending HTTP requests directly to uWSGI servers (start_stress_servers.sh) from python (stress.py). I needed to use two other computers to send the requests, since running it on the same machine was enough to hurt the server's performance, and one additional machine was not able to push the server to its limit.
Initially I ran into a problem where we are sending a user summary which, when the number of users get sufficiently high, uses more than our total available space for JSON in the sharedmemory protocol. We are sending this for a portion of the UI that really doesn't make sense for a group this large, so I turned it off for the rest of my testing.
With no sharding I measure a maximum of ~216 simulated clients, while with sharding I get ~1090.
Looking at CPU usage, the server process (python3) is at 73%, so still some headroom:
While it would be possible to make various parts more efficient and get even larger speed ups, I think this is will be sufficient for Solstice as long as we run it on a sufficiently parallel machine.
[1] All benchmarks in this post taken on the same 6processor
12thread Intel i78700 CPU @ 3.20GHz running Linux, courtesy of the
Swarthmore CS
department.
Comment via: facebook
Discuss
Notes on Wisdom
This post examines the virtue of wisdom. It is meant mostly as an exploration of what other people have learned about this virtue, rather than as me expressing my own opinions about it, though I’ve been selective about what I found interesting or credible, according to my own inclinations. I wrote this not as an expert on the topic, but as someone who wants to learn more about it. I hope it will be helpful to people who want to know more about this virtue and how to nurture it.
Singing the praises of wisdom at LessWrong has a bringing coals to Newcastle feel to it. After all, isn’t this community all about working hard and passionately to hack through the jungle of bias, illusion, and ignorance in search of the hidden temple of Athena?
So I was tempted to skip over wisdom and work on writing up some other virtue instead. But I’m hoping that by exploring wisdom asavirtue I can illuminate some facets of it that otherwise receive less attention here.
Two varieties of wisdomThere are two senses of wisdom that are found in some virtue traditions:
 phrónēsis, or “practical wisdom” (sometimes translated “prudence”), which concerns knowing how the world works, and reasoning well about how to pursue goals effectively (and about which goals are worth pursuing — which sometimes gets separated out into “conative wisdom”)
 philosophy, which concerns a more bigpicture understanding of “what it’s all about,” whether or not there seems to be any way to make practical use of that understanding
They are both important: Phrónēsis without philosophy can make you merely clever; while without phrónēsis, philosophy can leave you with your head in the clouds, unable to bring your wisdom down to earth where you can make it matter.
“The title wise is, for the most part, falsely applied. How can one be a wise man, if he does not know any better how to live than other men? — if he is only more cunning and intellectually subtle?” ―Thoreau
Philosophy is also sometimes considered an important end in itself. Aristotle thought it was the richest and most satisfying activity for people to engage in, and reasoned that it was the pastime of the gods.
The person with the virtue of wisdom habitually and regularly prioritizes thinking and behaving wisely. Which raises the question: why wouldn’t you? You might at first think that the only reason why you would deliberately think or behave unwisely is because you believe mistakenly that you are being wise.
That is one way you can go astray: you might understand the wise course of action based on the sort of situation you are in, but mistakenly believe you are in some other sort of situation; or viceversa, you might understand the situation you are in well enough, but be mistaken about how to confront situations of that sort wisely. But people are also deflected from wisdom by being overwhelmed by emotions like fear or anger, or by sensations like pleasure or pain. For this reason, virtues like courage, endurance, selfcontrol, and temperance can come to the assistance of wisdom.
Wisdom and mistakesIt is a popular belief that we gain wisdom (or gain it most effectively) by learning from our own mistakes.
“Wisdom is a virtue of old age, and it seems to come only to those who, when young, were neither wise nor prudent.” ―Hannah Arendt
On the other hand, learning from other people’s mistakes may be the more prudent way to go about it (#LFMF!). LessWrong is in part a collection of deadends marked by warning signs, pointing out the mistakes in reasoning that others have been waylayed by.
But you typically learn other people’s mistakes from other people’s failures, which may leave your own artisanal mistakes unchallenged. If you are willing to strap on your theories and go into battle with reality until you lose, you will be more likely to discover and shed your worst theories. This takes courage, confidence, industriousness, and a willingness to fail and to admit failure.
Surfing less unwisely“The fool doth think he is wise, but the wise man knows himself to be a fool.” ―Shakespeare (As You Like It)
At least since Socrates, wisdom has been associated with epistemic humility. The “LessWrong” name itself nods at that tradition: To be more wise, assume that you are wrong, try to figure out where and how, patch that up as best you can, lather, rinse, repeat. Don’t be too proud of the nuggets of wisdom you have dug up, but occasionally peer into the vast voids of ignorance, the blank spaces on the map. Imagine those things that could be true that would mean utterly overthrowing most of what you currently suspect to be true. Don’t become attached to your best guesses or too inclined to round off a high probability into a certitude, but always prefer reality to your favorite hypothesis.
Wisdom seems to have less to do with arriving on the firm ground of confident understanding, and more to do with learning to surf the unstable edge of profound uncertainty: neither clinging to the barelybuoyant flotsam of belief nor being pulled out into a sea of nihilism by a undertow of skepticism.
Mystical vs. rational wisdom techniquesTo understand and make our way in the world around us, we try to systematize, to find regularities, to discover causeandeffect relationships, and so forth. We create a map, using our knowledge of the territory that we have passed through, to help us predict the territory we are about to enter. By extrapolating from suggestive patterns in the world, our maps can illuminate things we do not experience directly, and can suggest places to look to discover more than we might have stumbled upon on our own. Habits of rational thinking help us to keep our maps from misrepresenting the territory, and warn us about where our maps might be misleading even when they are as accurate as we can make them.
Mystical wisdom techniques suggest a different way to go about it: rather than just improving your map and your mapreading, take some time also to look directly at the territory and improve the quality of your vision. The advantage of this approach is that you lose the compression artifacts and other errors that come from trying to reconstruct the territory from the map. A disadvantage is that while maps can sometimes be shared, visions have to be turned into maps before they can be — and by the time you have turned your vision into a map, there may be little to recommend it when compared with maps arrived at through more rational methods.
Discuss
The Darwin Game  Rounds 3 to 9
Rounds 39
MeasureBot maintains its lead. SimplePatternFinderBot takes second place.
Deep dive into SimplePatternFinderBot
SimplePatternFinderBotYonge's SimplePatternFinder can speak for itself.
if pattern != None: if pattern.IsPatternFairOrGoodForUs(): # Try and stick to it as it looks good ret = pattern.OurNext() else: # We have a problem. If it is a smart opponent we # don't want to encourage it to stick with it, on # the other hand if it is a dumb bot that will stick # with it regardless then we are better getting # something rather than nothing. It's also possible # we might not have been able to establish # cooperation yet if pattern.OurNext() >= 3: # The pattern is godd for us for at least this # move, so stick to it for now. ret = pattern.OurNext() elif (self.theirScore + pattern.GetNext())/self.turn\ >= 2.25: # Under no circumstances allow it to get too many # points from playing an unfavourable pattern ret = 3 elif self.theirMoves[1] + self.ourMoves[1] == 5: # If we managed to cooperate last round, # hope we can break the pattern and cooperate # this round. return self.theirMoves[1] elif not self.hasTotalOfFiveBeenPlayed: # If the combined scores have never been 5 # before try to arrange this to see if it will # break the deadlock. ret = pattern.OurNext() elif self.round < 4 and pattern.OurNext() >= 1: # It looks like we are probably dealing with # a nasty bot. Tolerate this within limits in # the early game where it is more likely to be # a dum bot than a sophisticated bot that is # very good at exploiting us, so we at least # get something ret = pattern.OurNext() elif self.round < 8 and pattern.OurNext() >= 2: # If we would get an extra point be tolerant # for a little longer. ret = pattern.OurNext() else: # It looks like it is being completly # unreasonable, so normally return 3 # to stop us from being exploited, # but occasionally offer 2 just in case # we have managed to accidentally get # ourselves into a defect cycle against # a more reasonable bot num = random.randint(0,50) if num == 0: # Possibly this should only be done once? ret = 2 else: ret = 3 Everything so far Today's Obituary Bot Team Summary Round Silly Chaos Bot NPCs Plays randomly. 4 Silly 4 Bot NPCs Always returns 4. 5 S_A Chaos Army "79% of the time it submits 1, 20% of the time it submits 5, 1% of the time it submits a random number between 0 and 5." 6 Silly 5 Bot NPCs Always returns 5. 6 Silly Invert Bot 0 NPCs Returns 0 onthe first round. Returns 5  <opponents_last_move> on subsequent rounds. 6 Silly 1 Bot NPCs Always returns 1. 6 PasswordBot Multics Fodder for EarlyBirdMimicBot 8 Definitely Not Collusion Bot Multics Fodder for EarlyBirdMimicBot 8 Silly Invert Bot 2 NPCs Returns 2 onthe first round. Returns 5  <opponents_last_move> on subsequent rounds. 9 Silly Random Invert Bot NPCs Plays randomly on first turn. Returns 5  <opponents_last_move> on subsequent rounds. 9 BenBot Norm Enforcers Collaborates with jacobjacob 9Rounds 1020 will be posted on November 16, at 5 pm Pacific Time.
Discuss
A SelfEmbedded Probabilistic Model
One possiblyconfusing point from the Embedded Agents sequence: it’s actually not difficult to write down a selfembedded world model. Just as lazy data structures can represent infinite sequences in finite space, a lazilyevaluated probabilistic model can represent a world which is larger than the data structure representing the model  including worlds in which that data structure is itself embedded. The catch is that queries on that model may not always be computable/welldefined, and even those which are computable/welldefined may take a long time to compute  e.g. more time than an agent has to make a decision.
In this post, we’ll see what this looks like with a probabilistic selfmodelling Turing machine. This is not the most elegant way to picture a selfembedded probabilistic model, nor the most elegant way to think about selfmodelling Turing machines, but it does make the connection from probabilistic models to quining and diagonalization explicit.
The ModelLet’s write out a Turing machine as a probabilistic model.
Pieces:
 Tape
 Head
 kbitpertimestep input channel
 kbitpertimestep output channel
 kbitpertimestep random bit channel
I’m including an input channel so we can have data come in at every timestep, rather than just putting all the input on one tape initially (which would let the machine perform arbitrary computation between each data point arriving). Similarly with the output channel: it will have a bit in it every timestep, so the machine can’t perform arbitrary computation between output bits. This is a significant difference between this model and the usual model, and it makes this model a lot more similar to realworld computational systems, like CPUs or brains or .... It is, effectively, a bounded computation model. This will play only a minor role for current purposes, but if you’re thinking about how decisiontheoretic queries work, it becomes much more relevant: the machine will only have finite time to answer a query before a decision must be made.
The relationships:
 The tapestate at time t.mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')} and position position, Tape(t)[position], is a function of Tape(t−1)[position] and Head(t) (if position=Head(t−1).position).
 The headstate at time t, Head(t), is a function of Head(t−1), tapestate at the head’s previous position Tape(t−1)[Head(t−1).position], and the input bits In(t) and Rand(t).
 The outputstate at time t, Out(t), is a function of Head(t).
Also, we’ll assume that we know the initial state head0 of the head and tape0 of the tape, and that only a finite section of the tape starts out nonzero. Put that all together into a probability distribution factorization, and we get
P[Tape,Head,OutIn,Rand]=I[Head(0)=head0]I[Tape[0]=tape0]⋅
∏tP[Head(t)Head(t−1),Tape(t−1)[Head(t−1).position],In(t),Rand(t)]⋅
∏tP[Tape(t)Tape(t−1),Head(t−1).position,Head(t)]⋅
∏tP[Out(t)Head(t)]
Each line here handles the update for one component  the head, tape and output  with the initial conditions on the first line. We can also further break apart the tapeupdate term, since only the tapeposition where the head is located depends on the headstate:
P[Tape(t)Tape(t−1),Head(t−1).position,Head(t)]=
P[Tape(t)[Head(t−1).position]Tape(t−1)[Head(t−1).position],Head(t)]⋅
∏position≠Head(t−1).positionP[Tape(t)[position]Tape(t−1)[position]]
Each of the "atomic" conditional probabilities in these formulas would be given by the machine's local updaterules  e.g. head state as a function of previous head state, state of previous tape location, and inputs, or tape state as a function of previous tapestate (usually the identity function). We can also incorporate a model of the rest of the world, which would give P[InOut] and P[Rand], with In at later times depending only on Out at earlier times so that the whole thing works out to a wellbehaved (i.e. acyclic) factorization.
The important thing to notice is that, while the distribution is over an infinite space, we can express it in a finite set of equations (i.e. the equations above). We can also perform ordinary probabilistic calculations using these equations: it’s just a plain old Bayes net, and there’s a bounded number of nonzero variables at each timestep. More to the point, we can hardcode a representation of these equations in the initial state of the Turing machine (quining to fit the representation of the initial state inside the representation of the model inside the initial state), and then the machine itself can perform ordinary probabilistic calculations using these equations. We can treat the equations defining the model as a lazy data structure, and run queries on them.
DiagonalizingSo, what happens if we try to diagonalize our selfmodelling Turing machine? What happens if we look at the first nonnull output of the machine and then flip it, and we program the machine to output the most likely value of this flipped output? Well, that scheme falls apart at “first nonnull output of the machine”. There’s no guarantee that the machine ever outputs a nonnull output. Let’s write this out concretely, in terms of the probabilistic model and queries involved.
We’ll assume that Out consists of two bits. The machine always outputs “00” until its computation completes, at which point it outputs either “10” if it wants to pass a logicalzero output or “11” for a logicalone output. The program on the initial tape is some ordinary probabilistic reasoning program, and we hardcode our query into it.
What should our query be? Well, we want to look at the first nonnull output, so we’ll have to write something like “Out(mintts.t.Out(t)≠“00”)”  the minimum time t such that the output is not "00". Then we want the machine to output the leastlikely value of that variable, so we’ll ask for something like
argminvalP[Out(mintts.t.Out(t)≠“00”)=“0”+val]
Then, when the machine goes to compute mintts.t.Out(t)≠“00”, the computation may not complete at all, in which case there will not be any time t which satisfies that condition.
The main thing to notice in this example is that the query itself was not welldefined. It’s not just that our machine can’t answer the query; even an outside observer with unlimited computational power would conclude that the answer is not welldefined, because the time t which it asks about does not exist. Our query is trying to address a variable which doesn’t exist. The “nonhalting weirdness” comes from passing in weird queries to the model, not from any weirdness in the model itself. If we stick to “normal” queries  i.e. ask about the value of a specific variable  then there isn’t any conceptual problem (though it may take a while for the query to run). So from this perspective, the central problem of selfembedded world models is not representation or interpretation of the model, but rather the algorithmic problem of expanding the set of queries we can answer “without any weirdness”.
In this example, there is another possible behavior: the machine may output a logical zero with probability ½ and a logical one with probability ½, using its random bit source. This would require a probabilistic reasoning algorithm quite different from what we normally use, but would be entirely selfconsistent and nonweird. That’s an example of what it might mean to “expand the set of queries we can answer without any weirdness”.
What Queries Do We Care About?We do not care about all queries equally. Depending on the details of our underlying model/logic, there may be lots of queries which are uncomputable/undefined, but which we don’t actually care about answering. We want a theory of embedded agents, but that does not necessarily imply that we need to handle every possible query in some very expressive logic.
So which queries do we need to handle, in order to support a minimum viable model of agency?
This is a hard question, because it depends on what decision theory we’re using, and exactly what kind of counterfactuals that decision theory contains (and of course some decision theories don’t directly use a probabilistic model at all). But there are at least some things we’ll definitely want  in particular, if we’re using a probabilistic model at all, we’ll want some way to do something like a Bayesian update. That doesn’t necessarily mean updating every probability of every state explicitly; we could update lazily, for instance, i.e. just store the input data directly and then go look it up if and when it’s actually relevant to a query. More generally, we want some data structure which summarizes whatever info we have from the inputs in a form suitable to answering whatever queries we’re interested in.
(To me, this sounds like an obvious usecase for abstraction: throw out info which is unlikely to be relevant to future queries of interest.)
Another interesting class of queries is optimization queries, of the sort needed for decision theories or gametheoretic agents. One way to think about the decision theoretic problems of embedded agency is that we want to figure out what the “right” class of queries is, in order to both (a) guarantee that the optimization query is actually solved solved correctly and quickly, and (b) get good performance in a wide variety of environments. (Of course, this isn’t the only way to think about it.)
Discuss
How to get the benefits of moving without moving (babble)
If you've been following along with the location discussion (you probably haven't, that's okay), you'll know that I've become convinced that trying to get the community to leave Berkeley en masse is probably not a good idea. However, that leaves us in a bit of a cheeky conundrum (sorry, been watching lots of British comedy) – there are in fact real reasons why some people are excited about moving, and we shouldn't just throw all that in the garbage, even if we decide not to move.
So in this post, I want to figure out how we can get the things that we want out of moving, without moving (thanks to Aray for the general idea). The point of this is to stop thinking of move/don'tmove as a binary, and instead focus on ways of achieving whatever goals are hidden at the root of our desire to move.
I'm choosing to focus on what I've come to believe are three of the main cruxes:
 Opportunity to stop stagnating / be a new person
 Political stability
 Nicer surroundings
I've taken inspiration from jacobjacob and generated 50 dumb ways to get each of the things (in spoiler tags, in case you want to generate your own!). In inviting you to do this babble challenge, I also invite you – if you so choose – to babble not on these topics, but on cruxes of your own.
Stop stagnatingWe've been in Berkeley for a long time, and some people just want to move because they want to be anywhere other than the place they already are. Your physical location definitely shapes the thoughts you have and the actions you take, so if you feel stuck in a rut, shaking up your whole life by moving can sound pretty appealing. How else can we shake up our lives?
Babble:
 Move to a different room in your house
 Start living with different housemates
 Move to a different physical house in the same neighborhood
 Move to a new neighborhood
 Rearrange your furniture
 Redecorate the house
 Paint your walls
 Spend a lot of time in VR
 Meditate a lot to become more attentive to your experience
 Start using a different room as the default common space
 Go to more conferences
 Go on retreats for much of the year
 Commute to an office instead of working from home, for context change
 Go on a walk / bike ride / drive in a different place each day
 Rotating offices  instead of having people from the same org working in the same place all the time, we reorganize once every one to three months
 Maybe we have one big office building and people work on different floors
 Maybe we keep the offices we have and just rotate the groups of people
 MIRI has a permanent retreat venue a couple hours away where researchers can go any time they want
 Go do more touristy things in the area where you live
 Go to more events
 Transition to a different gender
 Change the smellscape of your environment (e.g. with flowers, candles, or essential oils)
 Change the soundscape of your environment (e.g. by playing music all the time, or getting a lot of birds)
 Walk around on stilts or in high heels
 If you're bilingual, do all your workthinking in the other language
 Start learning a completely new field  e.g. art history for an AI researcher, or organic chemistry for a historian
 Get a dog
 Switch up your mode of transportation  e.g. if you usually bike everywhere, walk instead, or vice versa
 Make one of the rooms in your house a Dreamatorium
 Start sleeping in a tent in your yard
 Spend a night on the streets
 Don't have internet at your home, only at your office (or vice versa)
 Become nocturnal
 Read everything upside down
 Take drugs
 Take a monthlong vow of silence
 Sing everything you say
 Rhyme everything you say
 Call old friends you haven't talked to in years and ask for their take on the problems you're currently facing (whether personal or technical)
 Use lasers to make yourself colorblind
 Drastically switch up the aesthetic of your computerusing experience  e.g. by switching operating systems
 Start using a different web browser so that you get different kinds of results
 Get imprisoned
 Get rid of everything you own
 Have a baby
 Get married / divorced
 Implant electrodes in your brain
 Switch from typing your thoughts into a computer to writing them on paper
 Build a physical model of the abstract theory you're working on, e.g. out of wood or tinkertoys
 Take a job as a security guard or something, so you have a lot of time with nothing to do when you're not allowed to distract yourself with the internet, so you can have a bunch of unstructured thoughts
 Make new friends in a totally different social circle; their different way of thinking will help you generate new kinds of thoughts
 For organizations, have the ops team and researchers switch roles temporarily so that everyone can get a new perspective on the organization's goals.
 Completely revamp your routines – go back into explore mode for things where you've been in exploit mode a long time (e.g. restaurant choice, TV shows)
Whew, well, not all of those were completely useless! Onward!
Political and social stabilityA major thing lots of people want out of moving is to get away from the stressful uncertainty of recent social and political upheaval. How can we get that without moving?
Babble:
 Have people you trust run for public office
 Dedicate your life to founding a secret society that inserts people aligned with your values into positions of immense power in your country
 Singlehandedly disarm all the nukes in the world, like Superman in that one Justice League episode
 Buy all of the major news networks and let them mostly continue as they are but subtly make everything less partisan
 Write some very influential books
 Put sedatives in the municipal water supply
 Secede from your country / form a micronation
 Go really hard on raising the sanity waterline  e.g. get rationality training into all public schools
 Print ten million copies of HPMoR / the Sequences / SSC and distribute them evenly around your country
 Go back in time and finagle things so that there's less political polarization (not sure how, you figure it out)
 Find a Death Note and eliminate the people who are linchpins of social and political instability
 Like the previous one but in a technologically possible and yet still untraceable way, like… targeted asteroids
 Somehow become a big wig on Capitol Hill and spearhead some major bipartisan movement
 Invent a supervillaintype ray that causes all guns in the world to melt
 Become a Jesus / Gandhi / Forrest Gump type figure
 Purge Night
 Require mental health screenings for people before they can run for public office
 Abolish the CDC and FDA and most bureaucracy in the US; then people won't be angry because they had to wait five hours at the DMV and they won't be sick and angry about it because they can't afford healthcare
 Outlaw swear words
 Overhaul all of the algorithms that decide what to show people on the internet, to actively counter partisanship and general polarization
 Legalize marijuana and criminalize alcohol so that when people want to use legal drugs to numb their pain they're more likely to get chill than angry
 Get them vaccines distributed right quick like so we can end lockdown and therefore hopefully return to a better baseline of sanity
 Somehow import the collectivist values that make Japanese society so relatively functional
 Make the week into eight days instead, so that we get three rest days for every five workdays
 Make an AI that's a really great psychotherapist, then provide it for free to everyone in the world, and socially normalize or even require its use
 Positive singularity
 Build a giant fortress
 Get people to exercise more, because exercise is the magic that cures all ills
 Get people to make more art, because making art is the magic that cures all ills
 Automate away the vast majority of jobs and instead free people to make art or whatever, but also invent a fully immersive virtual reality experience (like Star Trek's holodecks) so that if they don't have anything productive they want to do they can just stay out of the way while being happy
 Just chill out, things are actually pretty fine
 More hugs
 Cause a whole lot more people who think like you (or the way you like people to think) to move to your area, a la Free State Project – then at least if shit goes down, you'll be surrounded by allies
 Become a citizen of another country, just in case
 Just really solidify your personal social group, and pretend people outside of your bubble don't exist
 Follow Eliezer's suggestions to reboot the police
 Make a society just like the one in Brave New World – that was a pisspoor attempt at a dystopia given that everyone is happy all the time, aging is curtailed, and society is incredibly stable
 Go hard on genetically modifying embryos so that within a generation everyone is smarter and more levelheaded
 Make many billions of dollars, take over the world
 Build a time machine, take over the world
 Nukes in space?, take over the world???
 Befriend a bunch of highly influential people (Bill Gates, Donald Trump, Kim JongUn, etc) and whisper in their ears like a vizier in a movie
 Replace a bunch of highly influential people with clones loyal to you
 Seduce Donald Trump
 Inundate the world with more resources than humanity could possibly use, so that there's nothing to fight over anymore. At the very least then we'd have different problems.
 Hire a whole team of bodyguards so you don't have to worry about violence
 Invent and spread widely a faster mode of travel, like hyperloop or flying cars, so that there's more global connectedness, and therefore maybe more global understanding
 Get the Autobots to come from Cybertron and save us from ourselves
 Join the military, rise in the ranks, take over the military, abolish the military
 Form a worldwide movement of people doing random acts of kindness – that's the kind of thing I thought might change the world when I was in high school, and who knows, it's not impossible
Finally, some people want to move because they just don't like the place they are all that much. I'm going to divide this babble in half, because there are two main classes of solutions: change your surroundings, or get better at accepting your surroundings as they are.
Changing your surroundings:
 Move to a nicer neighborhood in your area
 Move to a house with a big backyard
 Get a water feature
 Get lots of plants
 Redecorate your house
 Renovate your house
 Put a lot of effort into optimizing your work and living setups, your commute, etc.
 Become friends with everyone on your block, knock down the fences in your backyards, and make the area behind your houses a big private park
 Unilaterally shut down the street to car traffic and instead make it a place for kids to play
 Lobby for carfree roads or carfree days in your city
 Organize people to pick up litter in your neighborhood
 Generally combat the broken window effect in your neighborhood
 Fill your home with nice sounds and smells
 Plant a bunch of trees around your house
 Invent a way to replicate the effects of the Harry Potter noticemenot spell, so that most people can't perceive you, so you don't have to deal with them
 (WesternUSspecific) Fund controlled burns throughout the year all over the state to cap how bad wildfire season can get
 Secure all your furniture to the walls per earthquake best practices, so that you don't have to worry about things falling on you if there's a big earthquake
 If you don't like urban life, relocate to a suburb within commuting distance
 Abolish cars
 Make a zen garden
 VR
 KonMari your life
 Put a lumenator in every room where you spend time
 Decrease your exposure to your surroundings by staying home all the time (I bet a lot of you are already doing this :P) – then you only need to make your house good, which is way easier than making a whole city good
 Invent truly giant, like spaceshipsized air purifiers – they hover above the city and nullify all effects of pollution, wildfires, and even COVID
 Fill the air with happiness gas, like the Joker
 Exterminate all ticks / mosquitoes / whatever pest is the worst in your area

Accepting your surroundings:  Purposely go out in the world with a childlike sense of wonder – What kind of tree is that? Can you believe that cars exist? Can you believe that people exist?
 Mindfulness meditation
 See a therapist
 Have a conversation with someone who really likes living in the area
 Find a place in or near your home that you just genuinely love being, and soak up that feeling
 Start a gratitude journaling habit
 Remind yourself that the other places you've lived or might want to live aren't perfect; make a list of the ways in which those places aren't as good as your current city/area
 Remind yourself of the positive reasons that you initially ended up in your current location, and maybe try to get back some of that magic
 Be a tourist in your own city – benefit from all the very best things it has to offer
 Buy property so that you're locked into staying, and let posthoc justification work its magic
 Recite the Serenity Prayer
 Forbid yourself from saying or writing negative things about the place you live, so as to not strengthen those neural pathways
 Think of all the people you would never have met and things you would never have done if you hadn't been where you are
 Spend more time with your friends and be grateful that you live near them
 Think of all the ways you could have it worse – e.g., maybe Berkeley has some problems with being dirty, but it sure beats the slums of Mumbai
 Actually spend some time in a different place and think about all the things you miss about home
 Befriend a bunch of your neighbors
 Start participating in and organizing local community events so that you feel like you're a part of something nice
 Do community service to feel more connected to your city (not to purchase utilons, obv)
 Have kids, because kids need friends and need to go to school and stuff, which will cause you to become more integrated into the community
 Lobotomy
 Listen to Bobby McFerrin's Don't Worry Be Happy on loop until it sinks in. As a bonus you could buy one of those creepy fake fish to sing it for you.
 Post all over social media about how much you like the place you live – put #aesthetic pictures of it on Instagram, extoll its virtues on your Facebook, fight people on Twitter who don't like it. Eventually you will hopefully have convinced yourself you like the place, or at the very least, it will be too awkward of a social move to admit that you don't.
Well there you have it! I'd be interested to hear either other people's answers to these prompts, or their own cruxes. While I've largely made up my own mind on whether it's a good idea to move, I think most people still feel pretty unresolved. At the very least, it seems like there are real problems that need to be addressed – and if we don't move, we need to find other ways to address them.
Discuss
Interest survey: Forming an MIT Mystery Hunt team (Jan. 1518, 2021)
The upcoming MIT Mystery Hunt will be allremote, which presents an opportunity to form a widely distributed team for the LW/SSC community. The Mystery Hunt can be a lot of fun and is a great exercise in solving challenging problems. With enough people on the team we might even have a chance of winning!
If you think you might be interested please fill out the survey.
Discuss
Where can I find good explanations of the central limit theorems for people with a Bayesian background?
My knowledge of probability theory is based mostly on reading E.T. Jaynes’ Probability Theory book, Andrew Gelman’s blog, and various LessWrong posts. I now want to get a strong grasp of the central limit theorem(s), but YouTube videos and googled pages speak so much in the language of sampling from a population, and random variables, that it’s hard to be sure what they’re saying, given that my background doesn’t really include those ideas. I’m especially interested in the different kinds of CLTs, like the Lyapunov condition, the BerryEsseen theorem, and so on. I often have a tough time with diving right into algebra  something like http://personal.psu.edu/drh20/asymp/fall2002/lectures/ln04.pdf gives me terrible trouble. Given all these constraints, does anyone know of good resources from which I can gain a strong grasp of the CLTs?
Some things I am confused about after googling so far:
Do distributions converge to gaussians, or do means converge to the mean of a gaussian? Is the former a more difficult convergence to achieve, or are they actually the very same condition?
Is the CLT even about means? Does it say anything about the resulting variance or skewness of the resulting distribution?
Is it actually necessary to be sampling from a population, or does the CLT apply to taking the means of arbitrary distributions, regardless of where they were obtained?
Any form of media is OK, for recommendations  no preference. Please feel free to suggest things even if you’re not sure it’s what I’m looking for  you are probably better than google!
Discuss
A framework for thinking about single predictions
1. Motivation
It seems to me that:
 Predictions can be better or worse.
 This is true even with a sample size of 1.
 The quality of a prediction is not purely a function of its probability. (A 90% prediction for a false outcome may still be good, and two 70% predictions for the same outcome need not be equally good.)
 A lot of people have a hard time wrapping their heads around 13, and some reject them outright.
In this post, I work out a simple framework that justifies 13, then apply it to the 2020 election. (I argue that "was 538's prediction better than the markets' prediction?" is a coherent question whose answer is either 'yes' or 'no'.)
2. The FrameworkConsider the following game:
Throw a fair coin 100 times, counting the number of heads. You win the game if this number is at most 52 and lose if it is 53 or higher.Using the binomial formula, we can compute that the probability of winning this game is around 0.69135. (In mathy notation, that's P(X≤52).mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')} with X∼B(100,12).)
Suppose we play this game, writing down each coin flip as it occurs, and put the progression into a chart:
Here, the red line denotes the number of heads we have after n flips, and the blue line denotes the 'baseline', which is the number of heads such that continuing at this pace would end up at precisely 52 heads after 100 flips. We end the game above the baseline, with 54 heads, which means that we lose.
Since we know exactly how this game works, it's possible to compute the current probability of winning at every point during the game.[1] Here is a chart of these probabilities, given the flips from the chart above:
Note that the yaxis shows the probability of winning the game after observing the nth flip. Thus, there are precisely 101 values here, going from the probability after observing the 0th flip (i.e., without having seen any flips), which is ≈0.69135, to the probability after observing the 100th flip, which is 0. By comparing both graphs visually, you can verify that they fit together.
Each of these 101 yvalues is a prediction for the outcome of the game. However, they're evidently not equally wellinformed predictions: the ones further to the right are based on more information (more flips) than those further to the left, and this is true even for predictions that output the same probability. For example, we predict almost exactly 80% after both 13 and 51 flips, but the latter prediction has a lot more information to go on.
I call a graph like this an information chart. It tells us how the probability of a prediction changes as a function of {amount of input information}.
A separate aspect that also influences the quality of a prediction is calibration. In this case, all of the 101 predictions made by the blue curve are perfectly calibrated: if we ran the game a million times, took all 101 million predictions made by the one million blue curves, and put them all into bins, each bin would (very likely) have a proportion of true predictions that closely resembles its probability.[2]
However, while the blue curve has perfect calibration, we can also model imperfect calibration in this framework. To do this, I've computed the probability of winning, provided that we underestimate (red) or overestimate (green) the number of flips that are still missing, relative to those we know.[3] The results look like this:
Looking at these three information charts together, we can see the following:
 Whenever things look good, the red curve underestimates the remaining uncertainty and thus becomes too optimistic: it overshoots the blue curve.
 Whenever things look bad, the red curve still underestimates the remaining uncertainty, but now that means it becomes too pessimistic: it undershoots the blue curve.
 The green curve does the precise opposite: it always overestimates the remaining uncertainty and is thus less optimistic/less pessimistic than it should be.
Notably, 'good' and 'bad' have nothing to do with 50%. Instead, they are determined by the prior, the probability you would assign without any information. In this case, the prior is 0.69135, so the red curve overshoots the blue curve whenever the blue curve is above 0.69135 and undershoots the blue curve whenever the blue curve is below 0.69135. (And the green curve does the opposite in both cases.)
Unlike the blue curve, the red curve's predictions are not perfectly calibrated. Their calibration is pretty good around the 70% bin (because the prior 0.69135 happens to be close to 70%), but poor everywhere else. Predictions in the 90% bin would come true less than 9 out of 10 times, and predictions in the 50% bin would come true more than 5 out of 10 times. (And the opposite is true for the green curve.)
In summary, the following concepts fall out of this framework:
 Predictions can be based on more or less information.
 In expectation, more information moves the prediction in the right direction. However, every one piece of information may go in the wrong direction. We can visualize this relationship using an information chart.
 Predictions can have better or worse calibration.
 Calibration is about correctly estimating the remaining amount of uncertainty.[4]
My primary claim in this post is that information charts are the proper way to look at predictions – not just in cases where we know all of the factors, but always. It may be difficult or impossible to know what the information chart looks like in practice, but it exists and should be considered the philosophical ground truth. (Information isn't a onedimensional spectrum, but I don't think that matters all that much; more on that later.[5]) Doing so seems to resolve most philosophical problems.
I don't think it is in principle possible to weigh calibration and information against each other. Good calibration removes bias; more information moves predictions further away from the baseline. Depending on the use case, you may rather have a bunch of bold predictions that are slightly miscalibrated or a bunch of cautious predictions with perfect calibration. However, it does seem safe to say that:
 quality is a function of {amount of information} and {calibration}; and
 it increases monotonically with each coordinate. (I.e., all else equal, more information means a higher quality prediction, and better calibration means a higherquality prediction.)
Here is an example of where I think the framework helps resolve philosophical questions.
On the eve of the 2020 election, the model of 538 coded by Nate Silver has predicted an 89% probability of Biden winning a majority in the Electoral College (with 10% for Trump and 1% for a tie). At the same time, a weighted average of prediction markets had Biden at around 63% for becoming the next president.[6] At this point, we know that
 Biden has won; however
 there was a set of four states[7] such that, had the voting gap changed by ~0.7% in favor of Trump in all four, Trump would have won the Electoral College instead.
The first convenient assumption we will make here is that both predictions had perfect calibration. (This is arguably almost true.) Given that, the only remaining question is which prediction was made with more information.
To get my point across, it will be convenient to assume that we know how the information chart for this prediction looks like:
If you accept this, then there are two stories we can tell about 538 vs. Betting markets.[8]
Story 1: The Loyal Trump Voter
In the first story, noone had foreseen the real reasons why polls were off in favor of Trump; they may as well have been off in favor of Biden. Consequently, noone had good reasons to assign Biden a <89% chance of winning, and the people who did so anyway would have rejected their reasons if they had better information/were more rational.
If the above is true, it means everyone who bet on Biden, as some on LessWrong have advised, has taken a good deal. However, there is also a different story.
Story 2: The Efficient Market
In the second story, the markets knew something that modelers didn't: a 2016style polling error in the same direction was to be expected. They didn't know how large it would be exactly, but they knew it was large enough for 63% to be a better guess than 89% (and perhaps the implied odds by smart gamblers were even lower, and the price only came out 63% because people who trusted 538 bought overpriced Biden shares). The outcome we did get (a ~0.7% win for Biden) was among the most likely outcomes as projected by the market.
Alas, betting on Biden was a transaction which, from the perspective of someone knowing what the market knew, had negative expected returns.
In reality, there was probably at least a bit of both worlds going on, and the information chart may not be accurate in the first place. Given that, either of the two scenarios above may or may not describe the primary mechanism by which proTrump money entered the markets. However, even if you reject them both, the only specific claim I'm making here is that the election could have been such that the probability changed nonmonotonically with the amount of input information, i.e.:
 knowing more may have shifted the odds in one direction;
 knowing more still may have shifted them in the other;
 if so, both would have represented an improvement in the quality of the prediction.
If true, this yields a noninjective function, meaning that there are specific probabilities that have been implied by several positions on the information chart, such as the 63% for Biden. Because of this, we cannot infer the quality of the prediction solely based on the stated probability.
And yes: real information is not onedimensional. However, the principles still work in a 1000000dimensional space:
 The amount of information is still antiproportional to the distance between {current point} and {point of perfect information}. (Distances can be measured in arbitrarily highdimensional spaces.)
 Calibration is still about estimating this distance.
 If false information is included, the only thing that changes is that now there are points on the chart whose [distance to the point of full information] is larger than the distance between no information and full information.
Thus, to sum up this post, these are the claims I strongly believe to be true:
 One cannot go directly from "Biden won" to "therefore, 538's prediction was better since it had more probability mass on the correct outcome".
 One cannot go directly from "the election was close" to "therefore, the market's prediction was better since it implied more narrow margins".
And, perhaps most controversially:
 There is nothing in principle missing from the information chart picture. In particular, there is no meaningful way to say something like "even though it is true that gamblers who bet on Trump would not have done so if they were smarter/better informed, their prediction was still better than that of 538". If the premise is true (which it may not be), then the market's/538's predictions are analogous to two points on the blue curve in our coinflip game, with 538 being further to the right.[9]
I've basically said what I wanted to say at this point – the fourth chapter is there to overexplain/make more arguments.
One thing that falls out of this post's framework is that it makes sense to say that one prediction (and in extension, one probability) is better than another, but it doesn't make sense to talk about the correct probability – unless 'correct' is defined as the point of full information, in which case it is usually unattainable.
This also means that there are different ways to assign probabilities to a set of statements that are all perfectly calibrated. For example, consider the following eight charts that come from eight runs of the 100coins game:
There are many ways of obtaining a set of perfectly calibrated predictions from these graphs. The easiest is to throw away all information and go with the prior every time (which is the starting point on every graph). This yields eight predictions that all claim a 0.69135 chance of winning.[10] Alternatively, we can cut off each chart after the halfway point:
This gives us a set of eight predictions that have different probabilities from the first set, despite predicting the same thing – and yet, they are also perfectly calibrated. Again, unless we consider the point of full information, there is no 'correct' probability, and the same chart may feature a wide range of perfectly calibrated predictions for the same outcome.
You probably know that there are scoring functions for predictions. Our framework suggests that the quality of predictions is a function of {amount of information} and {calibration}, which begs the question of which of the two things scoring functions measure. (We would hope that they measure both.) What we can show is that, for logarithmic scoring,
 all else equal, better calibration leads to a better score; and
 assuming perfect calibration, more information leads to a better score.
The second of these properties implies that, the later we cut off our blue curves, the better a score the resulting set of predictions will obtain – in expectation.
Let's demonstrate both of these. First, the rule. Given a set of predictions p1,...,pn (with pi∈[0,1]) and a set of outcomes y1,...,yn (with yi∈{0,1}), logarithmic scoring assigns to this set the number
n∑i=1[yilog(pi)+(1−yi)log(1−pi)]
Since the yi are either 1 or 0, the formula amounts to summing up [the logarithms of the probability mass assigned to the true outcome] across all our predictions. E.g., if I make five 80% predictions and four of them come true, I sum up 4log(0.8)+log(0.2).
Note that these terms are all negative, so the highest possible score is the one with the smallest absolute value. Note also that log(x) converges to −∞ as x goes to 0: this corresponds to the fact that, if you increase your confidence in a prediction but are wrong, your punishment grows indefinitely. Predicting 0% for something that comes true yields a score of −∞.
I think the argument about calibration is the less interesting part (I don't imagine anyone is surprised that logarithmic scoring rewards good calibration), so I've relegated it into a footnote.[11]
Let's look at information. Under the assumption of perfect calibration, we know that any predictionwehaveassignedprobabilityp will, indeed, come true with probability p. Thus, the expected score for such a prediction[12] is
L(p):=plog(p)+(1−p)log(1−p)
We can plot L for p∈[0,1]. It looks like this:
This shows us that, for any one prediction, a more confident verdict is preferable, provided calibration is perfect. That alone does not answer our question. If we increase our amount of information – if we take points further to the right on our blue curves – some predictions will have increased probability, others will have decreased probability. You can verify this property by looking at some of the eight charts I've pictured above. What we can say is that
 in expectation, the sum of all probabilities remains constant (otherwise, either set of predictions would not be perfectly calibrated); and
 in expectation, our values move further apart (I don't have an easy argument to demonstrate this, but it seems intuitively obvious).
Thus, the question is whether moving away from p in both directions, such that the total probability mass remains constant, yields a higher score. In other words, we want that
2L(p)<L(p−ϵ)+L(p+ϵ)∀ϵ∈(0,min{p,1−p})
Fortunately, this inequality is immediate from the fact that L is strictly convex (which can be seen from the graph pictured above).[13] Similar things are true for the Brier score.[14]
I.e., the probability of having 52 heads total in 100 flips, conditional on the flips we've already seen. For example, the first two flips have come up tails in our case, so after flip #2, we're hoping for at most 52 heads in the next 98 flips. The probability for this is P(X≤52) with X∼B(98,12), which is about 76%. Similarly, we've had 12 heads and 10 tails after 22 flips, so after flip #22, we're hoping for at most 40 heads in the next 78 flips. The probability for this is P(X≤40) with X∼B(78,12), which is about 63%. ↩︎
To spell this out a bit more: we would run the game a million times and create a chart like the one I've shown for each game. Since each chart features points at 101 xpositions, we can consider these 101 million predictions about whether a game was won. We also know how to score these predictions since we know which games were won and which were lost. (For example, in our game, all predictions come out false since we lost the game.)
Then, we can take all predictions that assign a probability between 0.48 and 0.52 and put them into the '50%' bin. Ideally, around half of the predictions in this bin should come true – and this is, in fact, what will happen. As you make the bins smaller and increase the number of runs (go from a million to a billion etc.), the chance that the bins are wrong by at least ϵ converges to 0, for every value of ϵ∈R+.
All of the above is merely a formal way of saying that these predictions are perfectly calibrated. ↩︎
To be precise, what I've done is to take the function f(x)=x−x2100, which looks like this:
and use that to model how deluded each prediction is about the amount of information it has access to. I.e., after the 50th flip, the red curve assumes it has really seen 50+f(50)=75 flips, and that only 25 are missing. The value of those 75 flips is extrapolated from the 50, so if exactly half of the real 50 have come up heads, it assumes that 37.5 of the 75 have come up heads. In this case, this would increase its confidence that the final count of heads will be 52 or lower.
In general, after having seen n flips, the red curve assumes it knows n+f(n) flips. Since f(0)=0 and f(100)=0, it starts and ends at the same point as the blue curve. Similarly, the green curve assumes it knows n−f(n) many flips.
This may not be the best way to model overconfidence since it leads to 100% and 0% predictions. Then again, real people do that, too. ↩︎
It's worth pointing out that this makes calibration a property of single predictions, not an emergent property of sets of predictions. This seems like a good thing to me; if someone predicts a highly complex sequence of events with 50% probability, I generally don't feel that I require further predictions to judge its calibration. ↩︎
Furthermore, 'information' isn't restricted to the literal input data but is meant to be a catchall for everything that helps to predict something, including better statistical models or better intuitions. ↩︎
The outcomes for these predictions may come apart if someone who didn't win the election becomes president. (Although BetFair supposedly has ‘projected winner’ as its criterion.) ↩︎
That's Pennsylvania (~0.7% margin), Wisconsin (~0.7% margin), Georgia (~0.2% margin), and Arizona (~0.3% margin). ↩︎
Note that there are a lot more (and better) markets than PredictIt, I'm just using it in the image because it has a nice logo. ↩︎
To expand on this more: arguing that the market's prediction was better solely based on the implied margin seems to me to be logically incoherent:
 Biden won, so the naive view (which I strongly reject) says that higher probabilities were better.
 Deviating from the naive view implicitly assumes that confidently predicting a narrow win was too hard to be plausible, which is an argument about the information chart, unless the claim is that it's impossible due to quantum randomness. In particular, making this argument implicitly draws the 'implausible' region of the chart (i.e., the part that you don't believe anyone can enter) around just the final slope, so that betting 89%+ for good reasons was infeasible, but betting ~63% for good reasons was feasible. This may well be true, but making this argument acknowledges the existence of an information chart.
Incidentally, this set of eight predictions appears poorly calibrated: given their stated probability of 0.69135, we would expect about five and a half to come true (so 5 or 6 would be the good results), yet only 4 did. However, this is an artifact of our sample being small. Perfect calibration does not imply perfectlooking calibration on any fixed sample size; it only implies that the probability of apparent calibration being off by some fixed amount converges to zero as the sample size grows. ↩︎
Consider a set of predictions from the same bin, i.e., that we have assigned the same probability, p. Suppose their real frequency is p∗. We would now hope that the probability which maximizes our score is p∗. For each prediction in this bin, since it will come true with probability p∗, we will have probability p∗ to receive the score log(p) and probability 1−p∗ to receive the score log(1−p∗). In other words, our expected score is
p∗log(p)+(1−p∗)log(1−p)
To find out what value of p minimizes this function, we take the derivative:
p∗p−1−p∗1−p
This term is 0 iff p1−p=p∗1−p∗. Since the function ϕ(x)=x1−x is injective, we can apply ϕ−1 to both sides and obtain p=p∗ as the unique solution. Thus, calibration is indeed rewarded. ↩︎
It suffices to consider a single prediction since the scoring function is additive across predictions. ↩︎
Strict convexity says that, for all δ∈(0,1), we have
L(δx+[1−δ]y)<δL(x)+[1−δ]L(y)
Set δ=12 and x=p−ϵ and y=p+ϵ, then multiply the equation by 2. ↩︎
The Brier score measures negative squared distance to the outcomes, scaled by 1n. I.e., in the notation we've used for logarithmic scoring, we assign the number
−1nn∑i=1(pi−yi)2
The two properties we've verified for logarithmic scoring hold for the Brier score as well. Assuming perfect calibration, the expected Brier score for a prediction with probability p is −p(1−p)2−(1−p)p2. The corresponding graph looks like this:
Since this function is also strictly convex, the second property is immediate.
However, unlike logarithmic scoring, Brier score has bounded penalties. Predicting 0% for an outcome that occurs yields a score of −1 rather than −∞. ↩︎
Discuss
Страницы
 « первая
 ‹ предыдущая
 …
 2
 3
 4
 5
 6
 7
 8
 9
 10
 …
 следующая ›
 последняя »