Вы здесь
Новости LessWrong.com
Scope Insensitivity Judo
It's easy to bemoan scope insensitivity, a special case of that phenomenon where we mere humans end up caring more about the death of one person than one hundred, better remember the last bite of a meal than the first dozen, and think less is more and more is less. After all, if we didn't neglect scope we would be more rational, and so maybe happier and healthier, living in a world were everyone got more of what they wanted, since without scope insensitivity it wouldn't be so hard to convince people to help those far away who need more than those nearby who need less. But scope insensitivity is what we've got, so we have to learn to live with it.
Luckily there's plenty of reason to think we can take advantage of scope insensitive because people have already discovered ways to make the best of other forms of extension neglect. For example, adapting to duration neglect is something most people learn early on by adopting heuristics like "save the best for last" and "do the hardest part first". Salespersons and motivational speakers alike learn to exploit base rate neglect, sample size neglect, and the conjunction fallacy to convince people to do what they otherwise might not. And designers of all kinds of systems can mold incentives to work with rather than against human nature. Thus it stands to reason we can use our natural scope insensitivity to do more than fail to multiply.
I'll consider one such use case here, namely a practice of using scope insensitivity to prepare ourselves for highstakes situations in lowstakes ones. This is a kind of scope insensitivity judo, or "gentle way", and just like in the martial art, we'll redirect the strength of our "opponent" to transform it into an unintended ally.
Dwell in the DojoLife is full of highstakes situations: job interviews, first dates, nuclear missile crises. These generally feel like oneshot scenarios—there's one chance to get it right and if we fail all is lost. To wit, if we don't say the right things we'll lose our shot at that job forever, if we don't put out the right vibes that person will never fall in love with us, and if we push the big red button there'll be no second chances for anything.
We can make these multishot scenarios pretty easily with training, and there's some value in practicing interviewing skills, going on many dates so no one date matters very much, and running war games. These are all training methods that take something highstakes and make it lowstakes so you feel free to experiment. That's one way to learn: by creating a safe laboratory where we can explore more before we prune.
That's not what I'm suggesting we do, though. In the dojo of scope insensitivity judo, we practice the way of getting into lowstakes scenarios that feel highstakes so we are prepared generally to handle really highstakes scenarios when we encounter them. We do this by taking advantage of the way our minds mistakenly believe many lowstakes scenarios are highstakes ones because they push against beliefs and behaviors that were evolutionarily or historically adaptive but no longer are.
Consider these examples from my own life, drawn from my zen practice:
 I asked if I could bring a cushion from home for a retreat. I was told yes. I brought it. The cushion was orange, the zendo's cushions were black, it stuck out, and I was told I couldn't use my cushion.
 I complied, but I was immediately caught by thoughts like "but you told me I could use my cushion" and "now my meditation will be worse because I'll be less comfortable" and "I'm not as good a zen student as I thought".
 I felt embarrassed, defensive, let down, and defeated. I felt like a failure, like I was 2nd grade Gordy again getting in trouble for being weird.
 Of course, stepping back, we can see this was a very lowstakes situation: I just switched cushions and got on with the retreat! But it felt highstakes at the time because it pushed me in ways that might have been adaptive in some highstakes situations, either in my personal past or within my cultural or evolutionary environment. For our ancestors, this kind of mistake could have meant loss of prestige and thus loss of resources and thus marginal loss of reproductive and survival opportunities. Lucky for me it was just about a cushion in the zendo!
 I was sitting half lotus during a long meditation period, and after about 40 minutes my legs hurt in a way that I worried was injuring them by continuing, so I uncrossed my legs and sat with them pulled up towards my chest to give them a rest. In the middle of this my teacher walked into the zendo and saw me, and came over to correct me, saying I couldn't sit like that and had to either sit crosslegged or in a chair.
 I sort of complied: I instead took the option to do brief walking meditation before returning to sitting. I was caught by thoughts like "you didn't see how I had been sitting" and "you didn't know the kind of danger I was in" and "I must have stayed sitting outofform because the pain was so bad it temporarily addled my mind".
 I felt embarrassed, ashamed, and defensive and also a little indignant.
 This was also a pretty lowstakes situation: I walked for 10 minutes, came back and sat for the rest of the period, and it was never mentioned again. I didn't lose any of my positions or responsibilities, and my practice continued on as strong as ever. But it felt highstakes because I had been caught out and corrected in front of others, and maybe they would think less of me. As best I can tell, they did not.
 A new person came to our Saturday morning practice period for the first time. I was work leader that week, and when it came time to hand out assignments I assigned her to clean the zendo under my supervision. I was later corrected by the person who trained me as work leader that I shouldn't have given her that assignment because new people should get simple tasks like sweeping.
 I was immediately somewhat defensive. Cleaning the zendo was the job I had been assigned when I first came to the zen center, so I thought it was the right thing to do. I said as much.
 In addition to being defensive, I felt like I had been let down by my trainer not telling me this before, and I also felt I had the excuse that it worked out fine.
 Once again, this was pretty lowstakes: she cleaned the zendo well, I got new information, and I changed how I hand out work assignments. But my behavior indicates I thought it was highstakes enough to be worth some backandforth and argument or defense of my position and to put myself in opposition to another person to save face. I had wanted to do a good job at being work leader, and felt threatened by the correction, leading me to escalate my response.
I drew these from my zen practice because the zen center is like a laboratory where we specialize in studying the self, and so I had more chance to examine these events and remember them than the many similar daily occurrences that happen throughout the rest of my life. Also they are less personal and raw than the times I blew lowstakes situations outofproportion and didn't learn from them at work, with family, and among friends. But hopefully those are enough for you to start to see the pattern: scope insensitivity means we often treat lowstakes situations like highstakes situations, and we can take advantage of that to use them as training scenarios for genuine highstakes events if we allow ourselves the space to stop and take a step back to consider what we're doing.
The Way of Scope InsensitivityYou can begin to practice with scope insensitivity yourself right away, because the world is constantly presenting you with lowstakes scenarios that feel highstakes. The more anxious, depressed, or frustrated you generally are, the more you are likely you are treating lowstakes situations as highstakes and so you will have even more opportunities to practice scope insensitivity judo than people who are more calm and equanimous.
The first part of the practice is to notice and stop. Notice when you feel like you are in a highstakes situation. Then stop for a few breaths to examine it. Don't worry if you fail at first; learning to notice is hard if you're not already skilled at it, and even when you are skilled it's still easy to get so caught up that we forget to really look.
When you catch one of these situations, consider whether it is really highstakes, or if you just believe it is due to scope insensitivity. If it's really lowstakes, this is a great opportunity to experiment and practice with dealing with these situations and the factors that cause them to feel highstakes. If you're sure it's really highstakes, that's even better, though you'll want to be a bit more cautious in how you proceed.
There are many ways you can explore these situations once you've noticed them arising, and the path you take largely depends on what you are ready for and what resonates with you. I've gotten a lot of milage out of the Immunity to Change framework and working with core beliefs (albeit within the Ordinary Mind zen context rather than a CBT context). You might prefer something that looks more like psychotherapy, various CFAR techniques, Folding, Focusing, Core Transformation, or some kind of debugging. Generally you are looking for a way to integrate what you can see when you step back and look at what's happening in these situations with your immediate reactions, and anything that helps you do that will likely work here.
And then you just keep doing it. You're unlikely to fix your scope insensitivity—that appears to just be part of how human brains work. But you can, through regular practice, retrain yourself to more deftly handle situations that previously felt overwhelming. By developing the skill of flipping what feel like highstakes situations into lowstakes ones, you'll gain perspective on those situations that allows you to take a more thoughtful, deliberate approach that transcends the worst of our kneejerk reactions that lead to selfcreated suffering.
Crossposted to Map and Territory.
Discuss
From Laplace to BIC
The previous post outlined Laplace approximation, one of the most common tools used to approximate hairy probability integrals. In this post, we'll use Laplace approximation to derive the Bayesian Information Criterion (BIC), a popular complexity penalty method for comparing models with more free parameters to models with fewer free parameters.
The BIC is pretty simple:
 Start with the maximum log likelihood .mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')} lnP[dataθmax]
 Subtract k2lnN to penalize for complexity, where N is the number of data points and k is the number of free parameters (i.e. dimension of θ).
Thus: lnP[dataθmax]−k2lnN. Using this magic number, we compare any two models we like.
Let's derive that.
BIC DerivationAs usual, we'll start from P[datamodel]. (Caution: don't forget that what we really care about is P[modeldata]; we can jump to P[datamodel] only as long as our priors are close enough to be swamped by the evidence.) This time, we'll assume that we have N independent data points xi, all with the same unobserved parameters  e.g. N die rolls with the same unobserved biases. In that case, we have
P[datamodel]=∫θP[dataθ]dP[θ]=∫θ∏Ni=1P[xiθ]p[θ]dθ
Next, apply Laplace approximation and take the log.
lnP[datamodel]≈∑ilnP[xiθmax]+lnp[θmax]+k2ln(2π)−12lndet(H)
where the Hessian matrix H is given by
H=d2dθ2lnP[dataθ]θmax=∑id2dθ2lnP[xiθ]θmax
Now for the main trick: how does each term scale as the number of data points N increases?
 The max log likelihood ∑iP[xiθmax] is a sum over data points, so it should scale roughly proportionally to N.
 The prior density and the k2ln(2π) are constant with respect to N.
 H is another sum over data points, so it should also scale roughly proportionally to N.
Let's go ahead and write H as N∗(1NH), to pull out the Ndependence. Then, if we can remember how determinants scale:
lndet(N∗(1NH))=lndet(1NH)+k∗lnN
so we can rewrite our Laplace approximation as lnP[datamodel]≈∑ilnP[xiθmax]+p[θmax]+k2ln(2π)−12lndet(1NH)−k2lnN=lnP[dataθmax]−k2lnN+O(1)
where O(1) contains all the terms which are roughly constant with respect to N. The first two terms are the BIC.
In other words, the BIC is just the Laplace approximation, but ignoring all the terms which don't scale up as the number of data points increases.
When Does BIC Work?What conditions need to hold for BIC to work? Let's go back through the derivation and list out the key assumptions behind our approximations.
 First, in order to jump from P[modeldata] to P[datamodel], our models should have roughly similar prior probabilities P[model]  i.e. within a few orders of magnitude.
 Second, in order for any point approximation to work, the posterior parameter distribution needs to be pointy and unimodal  most of the posterior probability mass must be near θmax. In other words, we need enough data to get a precise estimate of the unobserved parameters.
 Third, we must have N large enough that k2lnN (the smallest term we're keeping) is much larger than the O(1) terms we're throwing away.
That last condition is the big one. BIC is a largeN approximation, so N needs to be large for it to work. How large? That depends how big lndet(1NH) is  N needs to be exponentially larger than that. We'll see an example in the next post.
Next post will talk more about relative advantages of BIC, Laplace, and exact calculation for comparing models. We'll see a concrete example of when BIC works/fails.
Discuss
Against Excessive Apologising
Many people would say that if you realise that you are in the wrong, then you should always apologise. Perhaps, they'd exclude sociopathic situations where this would be used to manipulate you, but that'd be it.
However, it's easy to forget that apologising creates a cost for the person who is apologised to. They have to read your message and perhaps write a reply. This later component is tricky if they aren't convinced that you've made up for it. It reminds them of an experience they might want to forget. Further, it requires them to deal with a topic they may be completely sick and tired of.
If you apologise, it should be because it helps prevent or mend a rift with the other person. You should be extremely cautious about apologising because that's what you think a nice person would do, as those are precisely the situations where you are likely to end up apologising with no benefit to anyone.
Discuss
[Link] Intro to causal inference by Michael Nielson (2012)
This is a link post for Michael Nielson's "If correlation doesn’t imply causation, then what does?" (2012).
I want to highlight the post for a few reasons:
(1) it is a wellwritten introduction by an experienced science communicator — Michael is an author of the most famous book on quantum computing;
(2) causal inference is an essential tool for understanding the world;
(3) two recent AI safety papers use causal influence diagrams to (a) understand agent incentives [arXiv, Medium] and (b) to provide a new perspective on some problems in AGI safety [arXiv].
Discuss
When does adding more people reliably make a system better?
Prediction markets have a remarkable property. They reward correct contrarianism. They incentivise people to disagree with the majority consensus, and be right. If you add more traders to a market, in expectation they price will be more accurate.
More traders means both more fish and more sharks.
(The movie "The Big Short" might be a very sad portrait of the global financial system. But it's still the case that a system in a bad equilibrium with deeply immoral consequences rewarded the outcasts who pointed out those consequences with billions of dollars. Even though socially, no one bothered listening to them, including the US Government who ignored requests by one of the fund managers to share his expertise about the events after the crash.)
Lots of things we care about don't have this property.
 Many social communities decline as more members join, and have to spend huge amounts of effort building institutions and rituals to prevent this.
 Many companies have their culture decline as they hire more, and have to spend an incredible amount of resources simply to prevent this (which is far from getting better as more people join). (E.g. big tech companies can probably have >=5 candidates spend >=10 hours in interviews for a a single position. And that's not counting the probably >=50 candidates for that position spending >=1h.)
 Online forums usually decline with growing user numbers (this happened to Reddit, HackerNews, as well as LessWrong 1.0).
In prediction markets the vetting process is really cheap. You might have to do some KYC, but mostly new people is great. This seems like a really imporant property for a system to have, and something we could learn from to build other such systems.
What other systems have this property?
Discuss
Becoming a Robust Person, or Organization
Epistemic Status – some mixture of:
 “My best guess, based on some theory, practice and observations. But very much _not_ battletested”
 but also, “poetry that’s designed to get an idea across that isn’t necessarily precisely accurate", intended to get across the generators for my current worldview.
 Was waiting to post this until I resolved some disagreements that seemed upstream of it, but I think it'll be awhile before that happens. idk. YOLO.
tl;dr:
People are not automatically robust agents, and neither are organizations.
An organization can become an agent (probably?) but only if it’s built right. Your default assumption should probably be that a given organization is not an agent (and therefore may not be able to credibly make certain kinds of commitments).
Your default assumption, if you’re building an organization, should probably be that it will not be an agent (and will have some pathologies common to organizations).
If you try on purpose to make it an agent, have good principles, etc…
...well, your organization probably still won’t be an agent, and some of those principles might get coopted by adversarial processes. But I think it’s possible for an organization to at least be better at robust agency (and, also better at being “good”, or “human value aligned”, or at least “aligned with the values of the person who founded it.”)
For a few years I’ve been crystallizing what it means to be a robust agent, by which I mean: “Reliably performing well, even if the environment around you changes. Have good policies. Have good meta policies. Be able to interface well with people who might have a wide variety of strategies, some of who might be malicious or confused.”
Becoming a robust personPeople are not born automatically strategic, nor are they born an “agent.”
If you want robust agency, you have to cultivate it on purpose.
I have a friend who solves a lot of problems using the multiagent paradigm. He spends a lot of effort integrating and empowering his subagents. He treats them like adults, makes sure they understand each other and trust each other. He makes sure each of them have accurate beliefs, and he tries to empower each of them as much as possible so they have no need to compete.
This… doesn’t actually work for me.
I’ve tried things like internal double crux or internal family systems, and so far, it’s just produced a confused “meh.” Insofar as “subagents” is a workable framework, I still have a pretty adversarial relationship with myself. (When I’m having trouble sleeping or staying off facebook, instead of figuring out what needs my subagents have and meeting them all... I just block facebook for 16 hours a day and program my computer to turn itself off every hour of the night starting at 11pm)
I'm tempted to write off my friend's claims as weirdposthocnarrative. But this friend is among the more impressive people I know, and consistently has good reasons for things that initially sound weird to me. (This shouldn't be strong evidence to you, but it's enough evidence for me personally to take it seriously)
I once asked him “so… how do you even get your subagents to say anything to each other? I can’t tell if I have subagents or not but if I do they sure seem incoherent. Have you always had coherent subagents?”
And he said (paraphrased by me), something like:
“You know how when you’re a baby, you’re a flailing incoherent mess. And then you become, like, a four year old and you can sort of communicate but you can’t keep promises or figure things out very well. And then you’re a teenager and… maybe you’re a reasonable person, but maybe you’re still angry and moody and think you know everything even though you’re like fourteenyearold and kinda insufferable? "But... eventually you become an actual person who can make reasonable trades, and keep contracts?“My subagents were like that. Initially they were incoherent like a baby. But I spent years cultivating them and teaching them and helping them grow and now they’re, like, coherent entities that have accurate beliefs and can negotiate with each other and it’s all super reasonable.”“An important element here was giving the subagents jobs. I looked at what Fear was doing, and one thing seemed to be “help me notice when a bad thing was going to happen to me.” And I said “Okay, Fear. This is now your official job. I will be helping you to do this. If you are doing a good job, or seem to be making mistakes, I will be giving you feedback about that.”This… was an interesting outlook.
The jury’s still out on whether subagents are a useful framework for me. But this still fit into an interesting metaframework.
Subagents or no, people don’t stop growing as agents when they become adults – there’s more to learn. I’ve worked over the past few years to improve my ability to think, and have good policies that defend my values while interfacing better with potential allies and enemies and confused bystanders.
I still have a lot more to go.
Becoming a robust organizationPeople are not automatically robust agents.
Neither are organizations.
Whether or not subagents are a valid frame for humans (or for particular humans), they seem like a pretty valid lens to examine organizations through.
An organization is born without a brain, and without a soul, and it will not have either unless you proactively build it one. And, I suspect, you are limited in your ability to build it one by the degree of soul and brain that you have cultivated in yourself. (Where “you” is “whoever is building the organization”, which might be one founder or multiple cofounders)
Vignettes of Organizational CoherenceEpistemic Status: Somewhat poetryesque. These vignettes from different organizations paint a picture more than they spell out an explicit argument. But I hope it helps express the overall worldview I currently hold.
Holding off on Hiring
YCombinator recommends that young startups avoid hiring people as long as possible. I think there are a number of reasons for this, but one guess is that you’re ability to grow the soul of your organization weakens dramatically as it scales. It’s much harder to communicate nuanced beliefs to manypeopleatonce than a few people.
The years where your organization is small, and everyone can easily talk to everyone… those are the years when you have the chance to plant the seed of agency and the spark of goodness, to ensure your organization grows into something that is aligned with your values.
The Human Alignment Problem
Ray Dalio, of Bridgewater, has a book of Principles that he endeavors to follow, and have Bridgewater follow. I disagree (or are quite skeptical about) a lot of his implementation details. But I think the metaprinciple of having principles is valuable. In particular, writing things down so that you can notice when you have violated your previously stated principles seems important.
One thing he talks a lot about is “getting in sync”, which he discusses in this blog post:
For an organization to be effective, the people who make it up must be aligned on many levels—from what their shared mission is, to how they will treat each other, to a more practical picture of who will do what when to achieve their goals. Yet alignment can never be taken for granted because people are wired so differently. We all see ourselves and the world in our own unique ways, so deciding what’s true and what to do about it takes constant work.Alignment is especially important in an idea meritocracy, so at Bridgewater we try to attain alignment consciously, continually, and systematically. We call this process of finding alignment “getting in sync,” and there are two primary ways it can go wrong: cases resulting from simple misunderstandings and those stemming from fundamental disagreements. Getting in sync is the process of openmindedly and assertively rectifying both types. Many people mistakenly believe that papering over diﬀerences is the easiest way to keep the peace. They couldn’t be more wrong. By avoiding conﬂicts one avoids resolving diﬀerences. People who suppress minor conﬂicts tend to have much bigger conﬂicts later on [...]While it is straightforward to have a meritocracy in activities in which there is clarity of relative abilities (because the results speak for themselves such as in sports, where the fastest runner wins the race), it is much harder in a creative environment (where diﬀerent points of view about what’s best have to be resolved). If they’re not, the process of sorting through disagreements and knowing who has the authority to decide quickly becomes chaotic. Sometimes people get angry or stuck; a conversation can easily wind up with two or more people spinning unproductively and unable to reach agreement on what to do.For these reasons, speciﬁc processes and procedures must be followed. Every party to the discussion must understand who has what rights and which procedures should be followed to move toward resolution. (We’ve also developed tools for helping do this). And everyone must understand the most fundamental principle for getting in sync, which is that people must be openminded and assertive at the same time.The Treacherous TurnThis particular description about the treacherous turn (typically as applied to AI, but in this case using the example of a human) feels relevant:
To master lying, a child should:1. Possess the necessary cognitive abilities to lie (for instance, by being able to say words or sentences).2. Understand that humans can (deliberately) say falsehoods about the world or their beliefs.3. Practice lying, allowing himself/herself to be punished if caught.If language acquisition flourishes when children are aged 1518 months, the proportion of them who lie (about peeking in a psychology study) goes from 30% at age 2, to 50% of threeyear olds, eventually reaching 80% at eight. Most importantly, they get better as they get older, going from blatant lies to pretending to be making reasonable/honest guesses. There is therefore a gap between the moment children could (in theory) lie (18 months) and the moment they can effectively lie and use this technique to their own advantage (8 years old). During this gap, parents can correct the kid's moral values through education.I’m not sure the metaphor quite holds. But it seems plausible that if you want an organization where individuals, teams and departments don’t lie (whether blatantly and maliciously, or through ‘honest goodhartesque mistakes’, or through something like Benquo’s 4levelsimulacrum concept), you have some window in which you can try to install a robust system of honesty, honor and integrity, before the system becomes too powerful to shape.
Sometimes bureaucracy is successfully protecting a thing, and that’s good
Samo's How to Use Bureaucracies matched my experience watching bureaucracies form. I’ve seen bureaucracies form that looked reasonably formedonpurposebyacompetentperson, and I’ve seen glimpses of ones that looked sort of cobbled together like spaghetti towers.
An interesting viewpoint I’ve heard recently is “usually when people are complaining that Bureaucracies don’t have souls, I think they’re just mad that the bureaucracy didn’t give them the resources they wanted. And the bureaucracy was specifically designed to stop from people like them from exploiting it.
“Academic bureaucracies, say, have a particular goal of educating people and doing research. If you come to them with a plan that will educate people or improve research, they will usually give you want you want. If you come to them trying to get weird special exceptions or faculties for saving the world or whatever, they’ll be like ‘um, our job is not to save the world, it is to educate people and do research. If we gave resources to every person with a pet cause, we’d fall apart immediately.'”“Likewise, if they impose a weird rule on you, it’s probably because in the past sometime fucked up in some way relating to that rule. And dealing with the fallout was really annoying, and they decided they didn’t want to have to deal with that fallout ever again. Sorry that you think you’re a good exception or the rule is stupid – part of the point of policies is to abstract away certain things so they can’t bother you and you can focus on what matters.”I’m not sure how often this is actually true and how often it’s just a convenient story (bureaucracies do seem to be built out of spaghetti towers). But it seems plausible in at least some cases. And it seems noteworthy that “having a soul” might be compatible with “include leviathanic institutions that don’t seem to care about you as a person.”
Sabotaging the Nazis
On the flipside...
LW user Lionhearted notes in Explicit and Implicit communication that during World War II, some allies went to infiltrate the Nazis and gum up the works. They received explicit instructions like:
“(11) General Interference with Organizations and Production [...](1) Insist on doing everything through “channels.” Never permit shortcuts to be taken in order to expedite decisions.(2) Make “speeches.” Talk as frequently as possible and at great length. Illustrate your “points” by long anecdotes and accounts of personal experiences. Never hesitate to make a few appropriate “patriotic” comments.(3) When possible, refer all matters to committees, for “further study and consideration.” Attempt to make the committees as large as possible — never less than five. [...](5) Haggle over precise wordings of communications, minutes, resolutions.(6) Refer back to matters decided upon at the last meeting and attempt to reopen the question of the advisability of that decision.(7) Advocate “caution.” Be “reasonable” and urge your fellowconferees to be “reasonable” and avoid haste which might result in embarrassments or difficulties later on.(8) Be worried about the propriety of any decision — raise the question of whether such action as is contemplated lies within the jurisdiction of the group or whether it might conflict with the policy of some higher echelon.”And... well, this all sure sounds like the pathologies I normally associate with bureaucracy. This sort of thing seems to happen by default, as an organization scales.
There's also Scott's IRB Nightmare.
Organizations have to make decisions and keep promises.
Why can’t you just have individual agents within an organization? Why does it matter that the organizationasawhole be an agent?
If you can’t make “real” decisions and keep commitments, you will be limited in your ability to engage in certain strategies, in some cases unable to engage in mutually beneficial trade.
Organizations control resources that are often beyond the control of a single person, and involve complicated decision making procedures. Sometimes the procedure is a legible, principled process. Sometimes a few key people in the roomwhereithappens hash things out, opaquely. Sometimes it’s a legiblebutspaghettitower bureaucracy.
Any of these can be fine. But it’s usually still something beyond the sumoftheindividual people involved.
Sometimes nobody has any power – everyone requires too many checks from too many other people and nothing gets done on purpose.
Sometimes you talk to the head of the org, and maybe you even trust the head of the org, and they say the org will do a thing, but somehow the org doesn’t end up doing the thing.
Sometimes, you can talk to each individual person at the org and they all agree Decision X would be best, but they’re all afraid to speak up because there isn’t common knowledge that they agree with Decision X. Or, they do all agree and know it, but they can’t say it publicly because The Public doesn’t understand Decision X.
So Decision X doesn’t get made.
Sometimes you talk to each individual person and they each individually agree that Decision X is good, and you talk to the entire group and the entire group seems to agree that Decision X would be good, but… somehow Decision X doesn’t get done.
I think it makes sense for bureaucracies to exist sometimes, and to have the explicit purpose of preventing people from exploiting things too easily. But, it’s still useful for some part of the institution to be able to make decisions and commitments that weren’t part of explicitlylaidout bureaucracy chain.
Porous movements aren’t and can’t be agents
I think that agency requires a membrane, something keeps particular people in and out, such that you have any deliberate culture, principles or decision making at all.
Relatedly, I think you need a membrane for Stag Hunts to work – if any rando can blunder into the formation at the last moment, there’s no way you can catch a stag.
Organizations have fairly strong membranes, and sometimes informal community institutions can as well. But this is relatively rare.
So while I’m disappointed sometimes when particular individuals and organizations don’t live up to the ideals I think they were trying for… I don’t think it makes much sense to hold most “movements” to the ideal of agency. Movements are too chaotic, too hard to police, too easy to show up in and start shouting and taking up attention.
Instead, instead, I think of movements as a place where a lot of people with similar ideals are clustered together. This makes it easier to find recruit people into organizations that do have membranes and can have principles.
Narrative control and contracts, as alternative coordination mechanisms
Another friend who ran an organization once remarked (paraphrased)
“It seemed like the organization’s main coordination mechanism was a particular narrative that people rallied around. When I was in charge, I felt like it was my job to uphold that narrative, even when the narrative got epistemically dicey. This felt really bad for my soul, and eventually I stopped being in charge.”“I’m not sure what to do about this problem – organizations need some kind of coordination mechanism. I think a potential solution might be to make central element of your company culture ‘upholding contracts.’ Maybe you don’t all share the same vision for the company, but you can make concrete trades. Some of those trades are “I will do X and you will pay me dollars”, and some might be between employees, like “I will work enthusiastically on this aspect of the company for 2 months if you work enthusiastically on that aspect of it.”This seems plausible to me. But importantly, I don’t think you get “uphold contracts” as a virtue for free. If you want your employees to be able to do it reliably, you need mechanisms to train and reinforce that. (I think if you recruit from some homogenous cultures it might come more automatically, but it’s not my default experience)
Integrity and AccountabilityHabryka recently wrote about Integrity and Accountability, and it seemed useful to just quote the summary here:
One lens to view integrity through is as an advanced form of honesty – “acting in accordance with your stated beliefs.”— To improve integrity, you can either try to bring your actions in line with your stated beliefs, or your stated beliefs in line with your actions, or reworking both at the same time. These options all have failure modes, but potential benefits.— People with power sometimes have incentives that systematically warp their ability to form accurate beliefs, and (correspondingly) to act with integrity.An important tool for maintaining integrity (in general, and in particular as you gain power) is to carefully think about what social environment and incentive structures you want for yourself.Choose carefully who, and how many people, you are accountable to:— Too many people, and you are limited in the complexity of the beliefs and actions that you can justify.— Too few people, too similar to you, and you won’t have enough opportunities for people to notice and point out what you’re doing wrong. You may also not end up with a strong enough coalition aligned with your principles to accomplish your goals.Open Problems in Robust Group AgencyExercises for the reader, and for me:
1. How do you make sure your group has any kind of agency at all, let alone be ‘valuealigned’
2. How do you choose people to be accountable to? What if you’re trying to do something really hard, and there seem to be few or zero people who you trust enough to be accountable to?
3. It seems like the last cluster of people who tried to solve accountability created committees and boards and bureaucracies, and… I dunno, maybe that stuff works fine if you do it right. But it seems easy to become dysfunctional in particular ways. What’s up with that?
3. What “rabbit” strategies are available, within and without organizations, that are selfreinforcing in the near term, that can help build trust, accountability, and robust agency?
4. What “stag” strategies could you successfully execute on if you had a small group of people working hard together?
4b. How can you get a small group of dedicated, aligned people?
5. How can people maintain accurate beliefs in the face of groupthink?
6. How can any of this scale?
Discuss
Thoughts on the 510 Problem
5 dollars is better than 10 dollars
The 510 Problem is a strange issue in which an agent reasoning about itself makes an obviously wrong choice.
Our agent faces a truly harrowing choice: it must decide between taking $5 (utility 5) or $10 (utility 10).
How will our agent solve this dilemna? First, it will spend some time looking for a proof that taking $5 is better than taking $10. If it can find one, it will take the $5. Otherwise, it will take the $10.
Fair enough, you think. Surely the agent will concede that it can't prove taking $5 is better than taking $10. Then, it will do the sensible thing and take the $10, right?
Wrong.
Our agent finds the following the following proof that taking $5 is better:
.mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')} F→(U=5)F→(¬F→(U=0))□((F→(U=5))∧(¬F→(U=0)))→F□((F→(U=5))∧(¬F→(U=0)))→((F→(U=5))∧(¬F→(U=0)))(F→(U=5))∧(¬F→(U=0))FLet's go over the proof.
Line 1: Taking $5 gives you $5.
Line 2: If F is true, then ~F>x is true for any x.
Line 3: If you find a proof that taking $5 gives you $5 and take $10 gives you $0, you'd take the $5.
Line 4: Combine the three previous lines
Line 5: Löb's Theorem
Line 6: Knowing that taking $5 gives you $5 and taking $10 gives you $0, you happily take the $5.
Simplified ExampleTo understand what went wrong, we'll consider a simpler example. Suppose you have a choice between drinking coffee (utility 1) and killing yourself (utility 100).
You decide to use the following algorithm: "if I can prove that I will kill myself, then I'll kill myself. Otherwise, I'll drink coffee".
And because a proof that you'll kill yourself, implies that you'll kill yourself, by Lob's Theorem, you will kill yourself.
Here, it is easier to see what went wrongproving that you'll kill yourself is not a good reason to kill yourself.
This is hidden in the original 510 problem. The first conditional is equivalent to "if I can prove I will take $5, then I'll take $5".
Hopefully, it's now more clear what went wrong. How can we fix it?
Solution?I once saw a comment suggesting that the agent instead reason about how a similar agent would act (I can't find it anymore, sorry). However, this notion was not formalized. I propose the following formalization:
We construct an agent .mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')} A. Each time A makes a decision, it increments an internal counter n, giving each decision a unique identity. A uses the following procedure to make decisions: for each action a, it considers the agent Aa,n. Aa,n is a copy of A (from when it was created), except that if Aa,n would make a decision with id n, it instead immediately takes action a. Then, if A can prove any of these agents has the maximum expected utility, it chooses the action corresponding to that agent.
Discuss
Prereq: Question Substitution
A quote from a 2002 paper by Kahneman and Frederick referencing some of Kahneman's earlier work with Tversky:
Early research on the representativeness and availability heuristics was guided by a simple and general hypothesis: when confronted with a difficult question people often answer an easier one instead, usually without being aware of the substitution.It's an interesting hypothesis, and even without looking at studies it seems plausible. I can think of conversations where I've been frustrated by the fact that the other person can't seem to actually answer the question I've asked them and keeps wiggling around. But it makes me wonder.
How would you tell the difference between A:
Brain receives questionQuestion is substituted for an easier one.
Brain replies with answer to easier question
and B:
Brain receives questionBrain replies using a heuristic that it applies to this sort of question.
(thanks to this comment for helping me notice the discrepancy)
Hmmm, it seems like they could easily produce the same answer. Does question substitution explain any more than "the brain uses heuristics"?
Yes and no. I read the first few pages of the paper, and pretty quickly noticed that Kahneman doesn't seem to think that a literal question substitution is happening. Most of the paper explores the question of "what governs what heuristic gets applied when?" and in other phrasings Kahneman states "[people reply to questions] as if she had been asked [other question]."
Despite the fact that literal question substitution might not happen that much, I can still get useful information by using the substitution frame. If I'm investigating whether or not I used a heuristic I wasn't aware of, I can ask myself: "Was I trying to answer an easier question? What are similar but simpler questions related to the one I was trying to answer? Are there any questions I know I'm good at answering that I might have substituted for the original one?"
You might even say that question substitution is a great... heuristic for figuring out if you were using a heuristic, in lieu of a more detailed model of how and when your brain picks heuristics. Yes, by all means read the rest of the paper and launch a quest into understanding in detail how the mind works. And while you're working on that, feel free to use the idea of question substitution to help you explore.
One thing that question substitution helps me notice is my tendency to "have a hammer and go looking for nails." When doing a substitution, what sort of other questions might you want to substitute in? Questions you're good at answering! This will be relevant later on when when exploring ways you can accidentally get stuck in never ending arguments.
Isn't this just...No. It's not. To be more explicit about what I was saying in cognitive fusion, I frequently see people take the following frame:
Sometimes, for no reason people make dumb mistakes, which they'd better fix as soon as I point them out."Well gee, there's got to be some reason people make these mistakes."
"Yeah, it's because they're dumb/irrational/unenlightened."
At best, telling someone not to make "dumb mistakes" let's them know how you are going to judge them. At worst, it asserts a world where "dumb mistakes" are this magical fundamental thing your brain spits out sometimes, and that you fix it by turning off the "dumb mistake" switch in your head. We all know where that is, right? Cool, just checking.
Thinking about question substitution orients me to the process of my brain swapping in a heuristic, instead of becoming fixated on the heuristic itself and how positively idiotic it is and how I can't believe anyone could possibly be dumb enough to use such a heuristic and I'm so glad that I don't use any heuristics that stupid...
And so it goes.
Fusion, substitution, and your journey into the mindI've been ignoring the second part of substitution, the part where you don't notice that you did a substitution. Hmmm, not noticing when something that feels like "just giving an answer" is actually composed of a multi step heuristic selection process. If I squint, this sort of looks like fusion. A very light sort of fusion, depending on how readily I go "Oh yeah, oops" when the substitution is pointed out.
"Wait, you seem to be diluting the meaning of fusion to refer to any sort of lack of awareness of what your mind is doing!"
Yeah, I'm definitely using fusion pretty broadly. I'm proposing that you can use fusion and substitution as two general lenses to explore how you're mind actually works. What is your mind doing, and how much are you aware of what it is doing? If you dig into ACT, you'll find that cognitive fusion is a richer concept with more backing than what I'm describing. If you read the paper I linked at he beginning, you'll find Kahneman and Frederick digging into all sorts of interesting mechanisms that govern how and when heuristics get applied.
I'm proposing thinking in terms of fusion and substitution as a first step out of "dumb mistakes" thinking. "I don't like the fact that my future is an empty hopeless void". Maybe I'm "just being dumb" and I should "get over it." Or maybe I substituted "Will there be anything good in my future?" with "Is there anything good in my life right now?" (because predicting the future is hard) and then fused to that (because it's a super emotionally charged low level thought and I've never trained in defusion). They both point to the same problem. Which one is easier to solve?
Discuss
The National Security Commission on Artificial Intelligence Wants You (to submit essays and articles on the future of government AI policy)
Laplace Approximation
The last couple posts compared some specific models for 20000 rolls of a die. This post will step back, and talk about more general theory for Bayesian model comparison.
The main problem is to calculate .mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')} P[datamodel] for some model. The model will typically give the probability of observed data x (e.g. die rolls) based on some unobserved parameter values θ (e.g. the p's in the last two posts), along with a prior distribution over θ. We then need to compute
P[datamodel]=∫θP[dataθ]dP[θ]
which will be a hairy highdimensional integral.
Some special model structures allow us to simplify the problem, typically by factoring the integral into a product of onedimensional integrals. But in general, we need some method for approximating these integrals.
The two most common approximation methods used in practice are Laplace approximation around the maximumlikelihood point, and MCMC (see e.g. here for application of MCMC to Bayes factors). We'll mainly talk about Laplace approximation here  in practice MCMC mostly works well in the same cases, assuming the unobserved parameters are continuous.
Laplace ApproximationHere's the idea of Laplace approximation. First, posterior distributions tend to be very pointy. This is mainly because independent probabilities multiply, so probabilities tend to scale exponentially with the number of data points. Think of the probabilities we calculated in the last two posts, with values like 10−70 or 10−20  that's the typical case. If we're integrating over a function with values like that, we can basically just pay attention to the region around the highest value  other regions will have exponentially small weight.
Laplace' trick is to use a secondorder approximation within that highvalued region. Specifically, since probabilities naturally live on a log scale, we'll take a second orderapproximation of the log likelihood around its maximum point. Thus:
∫θelnP[dataθ]dP[θ]≈∫θelnP[dataθmax]+12(θ−θmax)T(d2lnPdθ2θmax)(θ−θmax)dP[θ]
If we assume that the prior dP[θ] is uniform (i.e. dP[θ]=dθ), then this looks like a normal distribution on θ with mean θmax and variance given by the inverse Hessian matrix of the loglikelihood. (It turns out that, even for nonuniform dP[θ], we can just transform θ so that the prior looks uniform near θmax, and transform it back when we're done.) The result:
∫θelnP[dataθ]dP[θ]≈P[dataθmax]p[θmax](2π)k2det(−d2lnPdθ2θmax)−12
Let's walk through each of those pieces:
 P[dataθmax] is the usual maximum likelihood: the largest probability assigned to the data by any particular value of θ.
 p[θmax] is the prior probability density of the maximumlikelihood θ point.
 (2π)k2 is that annoying constant factor which shows up whenever we deal with normal distributions; k is the dimension of θ.
 det(−d2lnPdθ2θmax) is the determinant of the "Fisher information matrix"; it quantifies how wide or skinny the peak is.
A bit more detail on that last piece: intuitively, each eigenvalue of the Fisher information matrix tells us the approximate width of the peak in a particular direction. Since the matrix is the inverse variance (in one dimension 1σ2) of our approximate normal distribution, and "width" of the peak of a normal distribution corresponds to the standard deviation σ, we use an inverse square root (i.e. the power of −12) to extract a width from each eigenvalue. Then, to find how much volume the peak covers, we multiply together the widths along each direction  thus the determinant, which is the product of eigenvalues.
Why do we need eigenvalues? The diagram above shows the general idea: for the function shown, the two arrows would be eigenvectors of the Hessian d2lnPdθ2 at the peak. Under a secondorder approximation, these are principal axes of the function's level sets (the ellipses in the diagram). They are the natural directions along which to measure the width. The eigenvalue associated with each eigenvector tells us the width, and then taking their product (via the determinant) gives a volume. In the picture above, the determinant would be proportional to the volume of any of the ellipses.
Altogether, then, the Laplace approximation takes the height of the peak (i.e. P[dataθmax]p[θmax]) and multiplies by the volume of θspace which the peak occupies, based on a secondorder approximation of the likelihood around its peak.
Laplace Complexity PenaltyThe Laplace approximation contains our first example of an explicit complexity penalty.
The idea of a complexity penalty is that we first find the maximum log likelihood lnP[dataθmax], maybe add a term for our θprior lnp[θmax], and that's the "score" of our model. But more general models, with more free parameters, will always score higher, leading to overfit. To counterbalance that, we calculate some numerical penalty which is larger for more complex models (i.e. those with more free parameters) and subtract that penalty from the raw score.
In the case of Laplace approximation, a natural complexity penalty drops out as soon as we take the log of the approximation formula:
lnP[datamodel]≈lnP[dataθmax]+lnp[θmax]+k2ln(2π)−12lndet(−d2lnPdθ2θmax)
The last two terms are the complexity penalty. As we saw above, they give the (log) volume of the likelihood peak in θspace. The wider the peak, the larger the chunk of θspace which actually predicts the observed data.
There are two main problems with this complexity penalty:
 First, there's the usual issues with approximating a posterior distribution by looking at a single point. Multimodal distributions are a problem, insufficientlypointy distributions are a problem. These problems apply to basically any complexity penalty method.
 Second, although the log determinant of the Hessian can be computed via backpropagation and linear algebra, that computation takes O(k3). That's a lot better than the exponential time required for highdimensional integrals, but still too slow to be practical for largescale models with millions of parameters.
Historically, a third issue was the math/coding work involved in calculating a Hessian, but modern backprop tools like Tensorflow or autograd make that pretty easy; I expect in the next few years we'll see a lot more people using a Laplacebased complexity penalty directly. The O(k3) runtime remains a serious problem for largescale models, however, and that problem is unlikely to be solved any time soon: a lineartime method for computing the Hessian log determinant would yield an O(n2) matrix multiplication algorithm.
Discuss
Normalising utility as willingness to pay
.mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')}
I've thought of a framework that puts most of the methods of interteoretic utility normalisation and bargaining on the same footing. See this first post for a reminder of the different types of utility function normalisation.
Most of the normalisation techniques can be conceived of as a game with two outcomes, and each player can pay a certain amount of their utility to flip from one one outcome to another. Then we can use the maximal amount of utility they are willing to pay, as the common measuring stick for normalisation.
Consider for example the minmax normalisation: this assigns utility 0 to the expected utility if the agent makes the worst possible decisions, and 1 if they make the best possible ones.
So, if your utility function is u, the question is: how much utility would you be willing to pay to prevent your nemesis (a −u maximiser) from controlling the decision process, and let you take it over instead? Dividing u by that amount[1] will give you the minmax normalisation (up to the addition of a constant).
Now consider the meanmax normalisation. For this, the game is as follows: how much would you be willing to pay to prevent a policy from choosing randomly amongst the outcomes ("mean"), and let you take over the decision process instead?
Conversely, the mean minmean normalisation asks how much you would be willing to pay to prevent your nemesis from controlling the decision process, and shifting to a random process instead.
The mean difference method is a bit different: here, two outcomes are chosen at random, and you are asked now much you are willing to pay to shift from the worst outcome to the best. The expectation of that amount is used for normalisation.
The mutual Worth bargaining solution has a similar interpretation: how much would you be willing to pay to move from the default option, to one where you controlled all decisions?
A few normalisations don't seem to fit into the this framework, most especially those that depend on the square of the utility, such as variance normalisation or the Nash Bargaining solution. The Kalai–Smorodinsky bargaining solution uses a similar normalisation as the mutual worth bargaining solution, but chooses the outcome differently: if the default point is at the origin, it will pick the point (x,x) with largest x.
This, of course, would incentivise you to lie  but that problem is unavoidable in bargaining anyway. ↩︎
Discuss
Dialogue on Appeals to Consequences
Why it feels like everything is a tradeoff
Epistemic status: A cute insight that explains why it might feel like you always have to make sacrifices along one metric to get improvements along another. Seems tautological once you understand it. Might be obvious to everyone.
Metaepistemic status or something: My first post. Testing the waters.
Tl;dr: Skip to the last paragraph.
Example of a tradeoffI'm a programmer. I'm also a design prude. I'm also lazy. This all means that I spend a lot of my time chasing some different metrics in my code:
1) How easy it is to read.
2) How long it takes to run.
3) How long it takes to write.
4) ...
These metrics are often at odds with one another. Just the other day I had to make a tradeoff involving a function I'd written to evaluate a polynomial at a given point. Originally, it was written in a way that I felt was selfexplanatory: it looped over the derivativesatzero of the polynomial, which were passed in as a list, and summed up the appropriate multiples of powers of x — a Taylor sum. Pseudocode:
def apply_polynomial( deriv, x ):sum = 0
for i from 0 to length( deriv ):
sum += deriv[i] * pow(x, i) / factorial(i)
return sum
It turned out that this function was a significant bottleneck in the execution time of my program: about 20% of it was spent inside this function. I was reasonably sure that the pow and factorial functions were the issue. I also knew that this function would only ever be called with cubics and lowerdegree polynomials. So I rewrote the code as follows:
def apply_cubic( deriv, x ):sum = 0
len = length( deriv )
if len > 0:
sum += deriv[0]
if len > 1:
sum += deriv[1] * x
if len > 2:
square = x * x
sum += deriv[2] * square / 2
if len > 3:
cube = square * x
sum += deriv[3] * cube / 6
return sum
Sure enough, this improved the runtime significantly — by nearly the whole 20% that had been being spent inside this function. But notice that the code no longer contains the elements that define a Taylor sum: the loop is gone, and the factorials (0!, 1!, 2!, 3!) have been replaced with their values (1, 1, 2, 6). It also isn't obvious why the length comparisons stop at 3. The code no longer explains itself, and must be commented to be understood. Readability has been sacrificed on the altar of efficiency.
QuestionWhy am I cursed so? Why can't these metrics go handinhand? And in general, why am I always doing this sort of thing? Sacrificing flavor for nutrition in the cafeteria, sacrificing walkingspeed for politeness on a crowded sidewalk? Why are my goals so often set against one another?
AnswerLet's take a second to think about what a tradeoff is.
Suppose you're faced with a problem to solve. You have two solutions in mind (Solution A & Solution B), and you have two metrics (Metric 1 & Metric 2) by which to judge a solution. A tradeoff occurs when the solution that scores better along Metric 1 scores worse along Metric 2:
For example:
Of course, there need not be only two possible solutions. Maybe I'm willing to spend two hours working on improving this function, and depending on what I focus on, I could achieve any of the following balances:
And — argh! The correlation is negative! Why!
Well, there's a reason, and the reason is that this isn't the full picture. This is:
See, there are a whole bunch of ways to write the code that are neither as efficient nor as readable as one of the filledin circles on the perimeter. But I would never choose any of those ways, for obvious reasons. And if a new solution occurs to me that beats out some of my old solutions along both metrics...
...then this new solution would replace all the solutions strictly worse than it, which in turn would become part of the mass that resides below the curve:
No matter how many new solutions are introduced into the mix, and no matter by how far they outperform the old solutions, the outer frontier of nondominated solutions must have negative slope everywhere. A step to the right along this line must be accompanied by a step downward, because if it isn't, then the solution you just stepped off of is dominated by the one you just stepped onto, so the former wasn't on the line.
It doesn't take any math to generalize this result to situations where you have more than two metrics. Any solution that is dominated by another solution will be thrown out, so the options you end up considering form a set where no element dominates another, a.k.a. one where a gain along one metric entails a loss along at least one other, a.k.a. a tradeoff. Throwing solutions out is easy (and is sometimes done subconsciously when you're working in your domain of expertise, or done by other people before a decision gets to you), but weighing the remaining options usually takes some consideration. So, our subjective experience is that we spend most of our time and energy thinking about tradeoffs. Q.E.D.
Discuss
Prereq: Cognitive Fusion
In a post by Kaj Sotala, he introduces the very useful idea of cognitive fusion.
Cognitive fusion is a term from Acceptance and Commitment Therapy (ACT), which refers to a person “fusing together” with the content of a thought or emotion, so that the content is experienced as an objective fact about the world rather than as a mental construct. The most obvious example of this might be if you get really upset with someone else and become convinced that something was all their fault (even if you had actually done something blameworthy too).In this example, your anger isn’t letting you see clearly, and you can’t step back from your anger to question it, because you have become “fused together” with it and experience everything in terms of the anger’s internal logic.
You can become fused to an emotion, a voice in your head, a political view, and experience it to "just be true". I see this as a similar sort of fusion I hear musician talk about, where after years of practice their instrument begin to feel like a part of their body. They aren't "using their index finger to press the black note on a piano" they are "just playing G". This is analogous to being so caught up in your own anger that your partner is "just wrong and terrible" as opposed to "it sorta looks like you intentionally did something to annoy me and I'm worried about if you'll do this again in the future." (or whatever the actual case is)
Sometimes I think of there being a general fusion process where the brain collapses levels of inference. All of the steps that go into a given physical motion or thought process get compressed into a single dot. The thought process will be experienced as "just true" and the physical motion will be experience as an atomic action available to you. Sometimes you can "uncompress" the chain, and sometimes you can't.
Problems can arise when you fuse to a thought or emotion that doesn't have an accurate view of the world, and you unknowingly take it's broken map as the territory.
Isn't this just "Don't make assumptions"?Not quite, though it is similar. Assumptions don't really capture the more general fusing process that you can also see with physical movement. "I can't believe that you just assumed you start off on your left foot when making a layup!" Nah, doesn't feel right. But the main reason I prefer to talk in terms of fusion is that "fusion" makes me focus on the process of attaching to something, while "assumptions" makes me focus on the object being attached to.
It's easier to see this difference when the though being fused to (or the claim being assumed) is "obviously" wrong, or at least obvious to one who isn't fused to it. The assumption frame makes me feel like my work is done when I find the other persons "dumb" assumption. Point it out with a pithy "Checkmate [outgroup]" and move on. The fusion frame leads me to ask "How did they get fused to this in the first place? How might I help them defuse from it?" By focusing on the process of attachment (fusion) I can appreciate how common it is to fuse to something and how hard it can be to defuse. When I focus on the object of attachment (assumption) I'm mostly thinking about just how stupid it is and how I can't believe that anyone would be dumb enough to fall for this, and I most certainly don't believe anything that stupid....
And so it goes. Thinking of some behavior as a "dumb mistake" makes you more likely to not notice when you engage in it. Some thoughts take moments to defuse from. Others take a lifetime. Sometimes you fuse to things, and you'd be wise to learn how it works rather than to ridicule it.
As you may have guessed, later parts of this sequence will talk about what can happen when you fuse with language. For now, just remember what fusion is, and treat it with respect.
"Modern man can't see God because he doesn't look low enough." Carl Jung
Discuss
Why did we wait so long for the bicycle?
h/t alyssavance
The bicycle, as we know it today, was not invented until the late 1800s. Yet it was a simple mechanical invention. It would seem to require no brilliant inventive insight, and certainly no scientific background.Why, then, wasn’t it invented much earlier?I asked this question on Twitter, and read some discussion on Quora. People proposed many hypotheses, including:+ Technology factors. Metalworking improved a lot in the 1800s: we got improved iron refining and eventually cheap steel, better processes for shaping metal, and ability to make parts like hollow tubes. Wheel technology improved: wirespoke (aka tensionspoked) wheels replaced heavier designs; vulcanized rubber (1839) was needed for tires; inflatable tires weren’t invented until 1887. Chains, gears, and ball bearings are all crucial parts that require advanced manufacturing techniques for precision and cost.+ Design iteration. Early bicycles were inconvenient and dangerous. The first version didn’t even have pedals. Some versions didn’t have steering, and could only be turned by leaning. (!) The famous “pennyfarthing” design, with its huge front wheel, made it impossible to balance with your feet, was prone to tipping forward on a hard stop, and generally left the rider high in the air, all of which increased risk of injury. It took decades of iteration to get to a successful bicycle model.+ Quality of roads. Roads in the 1800s and earlier were terrible by modern standards. Roads were often dirt, rutted from the passage of many carts, turning muddy in the rain. Macadam paving, which gave smooth surfaces to roads, wasn’t invented until about 1820. City roads at the time were paved with cobblestones, which were good for horses but too bumpy for bicycles. (The unevenness was apparently a feature, assisting in the runoff of sewage—leading one Quora answer to claim that the construction of city sewers was what opened the door to bicycles.)+ Competition from horses. Horses were a common and accepted mode of transportation at the time. They could deal with all kinds of roads. They could carry heavy loads. Who then needs a bicycle? In this connection, it has been claimed that the bicycle was invented in response to food shortages due to the “Year without a Summer”, an 1816 weather event caused by the volcanic explosion of Mt. Tambora the year earlier, which darkened skies and lowered temperatures in many parts of the world. The agricultural crisis caused horses as well as people to starve, which led to some horses being slaughtered for food, and made the remaining ones more expensive to feed. This could have motivated the search for alternatives.+ General economic growth. Multiple commenters pointed out the need for a middle class to provide demand for such an invention. If all you have are a lot of poor peasants and a few aristocrats (who, by the way, have horses, carriages, and drivers), there isn’t much of a market for bicycles. This is more plausible when you realize that bicycles were more of a hobby for entertainment before they became a practical means of transportation.+ Cultural factors. Maybe there was just a general lack of interest in useful mechanical inventions until a certain point in history? But when did this change, and why?These are all good hypotheses. But some of them start to buckle under pressure:The quality of roads is relevant, but not really the answer. Bicycles can be ridden on dirt roads or sidewalks (although the latter led to runins with pedestrians and made bicycles unpopular among the public at first). And historically, roads didn’t improve until afterbicycles became common—indeed it seems that it was in part the cyclists who called for the improvement of roads.I don’t think horses explain it either. A bicycle, from what I’ve read, was cheaper to buy than a horse, and it was certainly cheaper to maintain (if nothing else, you don’t have to feed a bicycle). And it turns out that inventors were interested in the problem of humanpowered vehicles, dispensing with the need for horses, for a long time before the modern bicycle. Even Karl von Drais, who invented the first twowheeled humanpowered vehicle after the Year without a Summer, had been working on the problem for years before that.Technology factors are more convincing to me. They may have been necessary for bicycles to become practical and cheap enough to take off. But they weren’t needed for early experimentation. Frames can be built of wood. Wheels can be rimmed with metal. Gears can be omitted. Chains can be replaced with belts; some early designs even used treadles instead of pedals, and at least one design drove the wheels with levers, as on a steam locomotive.So what’s the real explanation?(Continue reading. 2,184 words and lots of great bicycle pictures.)
This post is a single piece from Jason Crawford's project, The Roots of Progress, aptly named, to understand the nature and causes of human progress. I haven't thought deeply enough to check his research, but it's a fascinating project. This essay examines a specific piece of technology, but the case study is used to develop and support models of what it takes for progress to occur.
Discuss
1st Athena Rationality Workshop  Retrospective
During June 710 we ran the first Athena Rationality Workshop at the EA Hotel. The workshop taught the Ease process, which is a mental self debugging framework developed by Matt Goldenberg. The Ease framework incorporates various things he learned and developed during his 5 years of working as a personal coach.
The Ease process is a step by step method going through
 Awareness  noticing you thoughts, feelings and other mental objects
 Introspection  finding the root cause of what is going on in your mind
 Acceptance  accepting that you are who you are in the present and the choices you have done in the past
 Alignment  creating peace between your internal parts that are involved in the issue you are trying to fix
 Ecology  checking that whatever shift you made in you mind during the alignment step, does not create new problems elsewhere
 Creation  building the attitude, habits and external support that you need to keep on track with your new solution
 Integration  deeply integrate and reinforce new habits and attitude into your mind
Most of the material was taught by Matt. Ryan Thomas, Deni Pop, Toon Alfrink and Linda Linsefors (me) gave one or two lectures each and, to different extents, helped participants during exercises.
Overall the workshop went really well. We got several pieces of positive feedback during the workshop, and from the feedback form as well, although there is room for improvement (see further down). On average the participants estimated the time spent at the workshop to 9.5 times more productive compared to what they would have done otherwise, with answers ranging from 0.5 to 50.
"I was very impressed by the quality of the workshop; to be able to string together so many disparate theories, tools, and techniques into a single coherent framework, is nothing short of genius. Brilliant work!"  Olivier Maas (participant)
Who we areMatt Goldenberg has been working intensely on selfimprovement for the past 15 years, was a professional coach for 5 years and has been running applied rationality group workshops for the past 18 months.
Linda Linsefors organizes workshops. She gets the job done.
Toon Alfrink firmly decided to turn around his mental health issues once and for all. Somehow, this time, it actually worked. He visited a Zen Monastery for 3 months and hasn’t been depressed ever since. His productivity benefited just as much, going from struggling at half speed in uni to running an AI Safety startup, the LW Netherlands community, and getting good grades in uni all at once. To the extent that he has an idea how the hell all of this happened, he would like to teach you.
Denisa Pop was a licensed cognitivebehavioral therapist for 5 years and has a PhD in positive humananimal interaction. Now she uses her previous knowledge to improve humanhuman interaction, doing research in rational compassion and being a Community Manager at the EA hotel.
Ryan Thomas is a student of mindfulness and cognitive psychology. He’s spent the last several years traveling in order to learn and practice techniques for resolving internal conflict and solidifying self alignment.
Things that could have gone betterThe first Athena Rationality Workshop was definitely rushed in several ways. We started planning the workshop just a few weeks before the event which was not enough time. The schedule was finalized only the day before the workshop.
If we had started planning earlier we would have had time to notice that the schedule was way too full and time to figure out which parts would be OK to cut out which out hurting the rest of the program. Some lessons were cut out during the workshop, but mostly the result of this mistake was that the workshop itself was rushed too. The participants did not get enough time to practice the techniques or to rest and digest what had been learned.
Another mistake that can also be traced to not having enough preparation time, is that some of us who were supposed to be around as mentors during exercises (me among others) were badly prepared. For me about two thirds of the content of the workshop was new, so I ended up spending most of the workshop just learning the stuff for myself, rather than helping others.
How to do this better next timeThe reason the workshop planning was rushed is because we wanted to do it before Matt flew back to the US. Buy the time we decided to run the workshop, we could do it fast or not at all. This will not be a problem for the next workshop because we have already started the preparations.
As for Matt’s location, given the outcome of the first workshop, we are more secure in the value of the Athena Rationality Workshop, which means we can charge enough money for the event, in order to just fly Matt over when we need him.
Based on the feedback from the first Athena Rationality Workshop, we are going to cut down the content to only include the most useful and essential parts. The next workshop will be both shorter and less intense. (We are also discussing developing a more advanced week long version of the workshop, but that will take a bit longer.)
Before the next workshop, we also plan to have a few days of mentorship training, so that everyone who is mentoring during the workshop knows all the techniques, and knows what to do during the workshop.
The Next Athena Rationality WorkshopWe are planning to run the next Athena Rationality Workshop some time in October or November, possibly adjacent to EA Global in London. If you want to participate, please fill in our interest form.
Online versionWe are also developing an online version and are currently looking for alpha testers. The commitment would be 30 minutes a day for 7 days, each day practicing a new skill related to dealing with procrastination and akrasia. Click here for more info and signup.
Discuss
Wolf's Dice II: What Asymmetry?
In the previous post, we looked at Rudolph Wolf's data on 20000 rolls of a pair of dice. Specifically, we looked at the data on the white die, and found that it was definitely biased. This raises an interesting question: what biases, specifically, were present? In particular, can we say anything about the physical asymmetry of the die? Jaynes addressed this exact question; we will test some of his models here.
Elongated Cube ModelsJaynes suggests that, if the die were machined, then it would be pretty easy to first cut an even square along two dimensions. But the cut in the third dimension would be more difficult; getting the length to match the other two dimensions would be tricky. Based on this, we'd expect to see an asymmetry which gives two opposite faces (1 & 6, 2 & 5, or 3 & 4) different probabilities from all the other faces.
Here's what the model looks like for the 1 & 6 pair:
 1 & 6 each have the same probability .mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')} p2
 2, 3, 4 & 5 each have the same probability 1−p4
 Uniform prior on .mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')} p (i.e. dirichlet with α=1)
Let's call this model1,6.
I will omit the details of calculations in this post; readers are welcome to use them as exercises. (All the integrals can be evaluated using the dirichletmultinomial α=1 formula from the previous post.) In this case, we find
P[datamodel1,6]=n!1!(n+1)!((n1+n6)!n1!n6!(12)n1+n6)((n2+...+n5)!n2!…n5!(14)n2+n3+n4+n5)≈2.2∗10−59
For the other two opposite face pairs, we get:
 2,5: 1.4∗10−63
 3,4: 8.5∗10−29
... sure enough, an asymmetry on the 3,4 axis goes a very long way toward explaining this data.
Recall from the previous post that the unbiased model gave a marginal likelihood P[datamodel] around 10−70, and the biased model with separate probabilities for each face gave around 10−20. So based on the data, our 3,4 model is still about a billion times less probable than the full biased model (assuming comparable prior probabilities for the two models), but it's getting relatively close  probabilities naturally live on a log scale. It looks like the 34 asymmetry is the main asymmetry in the data, but some other smaller asymmetry must also be significant.
Just for kicks, I tried a model with a different probability for each pair of faces, again with uniform prior on the p's. That one came out to 1.7∗10−30  somewhat worse than the 3,4 model. If you're used to traditional statistics, this may come as a surprise: how can a strictly more general model have lower marginal likelihood P[datamodel]? The answer is that, in traditional statistics, we'd be looking for the unobserved parameter values p with the maximum likelihood P[datamodel,p]  of course a strictly more general model will have a maximum likelihood value at least as high. But when computing P[datamodel], we're integrating over the unobserved parameters p. A more general model has more ways to be wrong; unless it's capturing some important phenomenon, a smaller fraction of the parameter space will have high P[datamodel,p]. We'll come back to this again later in the sequence.
Pip Asymmetry ModelJaynes' other main suggestion was that the pips on the die are asymmetric  i.e. there's less mass near the 6 face than the 1 face, because more pips have been dug out of the 6 face.
As a first approximation to this, let's consider just the asymmetry between 1 and 6  the pair with the highest pip difference. We'll also keep all the structure from the 3,4 model, since that seems to be the main asymmetry. Here's the model:
 3 & 4 have the same probability p2 , as before
 2 & 5 have the same probability 1−p4, as before
 1 & 6 together have probability 1−p2, same as 3 and 5 together, but their individual probabilities may be different. Conditional on rolling either a 1 or 6, 1 comes up with probability p′ and 6 with probability (1−p′)
 Both p and p′ have uniform priors
The conditional parameterization for 1 & 6 is chosen to make the math clean.
Let's call this model3,4+pip. Marginal likelihood:
P[datamodel3,4+pip]=n!1!(n+1)!((n3+n4)!n3!n4!(12)n3+n4)∗((n1+n2+n5+n6)!(n1+n6)!n2!n5!(12)n1+n6(14)n2+n5)((n1+n6)!1!(n1+n6+1)!)≈2.3∗10−16
... and now we have a model which solidly beats separate probabilities for each face!
(I also tried a pip model by itself, without the 3,4 asymmetry. That one wound up at 2.1∗10−70  almost as bad as the full unbiased model.)
We can also go one step further, and assume that the pip difference also causes 2 and 5 to have slightly different probabilities. This model gives P[datamodel]≈3.9∗10−17  a bit lower than the model above, but close enough that it still gets significant posterior probability (about 3.9∗10−173.9∗10−17+2.3∗10−16=14% assuming equal priors; all the other models we've seen have nearzero posterior assuming equal priors). So based on the data, the model with just the 16 pip difference is a bit better, but we're not entirely sure. My guess is that a fancier model could significantly beat both of these by predicting that the effect of a pip difference scales with the number of pips, rather than just using whole separate parameters for the 16 and 25 differences. But that would get into hairier math, so I'm not going to do it here.
To recap, here's what model3,4+pip says:
 3 and 4 have the same probability, but that probability may be different from everything else
 2 and 5 have the same probability, and 1 and 6 together have the same probability as 2 and 5, but 1 and 6 have different probabilities.
That's it; just two "free parameters". Note that the full biased model, with different probabilities for each face, is strictly more general than this  any face probabilities p which are compatible with model3,4+pip are also compatible with the full biased model. But the full biased model is compatible with any face probabilities p; model3,4+pip is not compatible with all possible p's. So if we see data which matches the p's compatible with model3,4+pip, then that must push up our posterior for model3,4+pip relative to the full unbiased model  model3,4+pip makes a stronger prediction, so it gets more credit when it's right. The result: less flexible models which are consistent with the data will get higher posterior probability. The "complexity penalty" is not explicit, but implicit: it's just a natural consequence of conservation of expected evidence.
Next post we'll talk about approximation methods for hairy integrals, and then we'll connect all this to some common methods for scoring models.
Discuss
Nutrition heuristic: Cycle healthy options
It also doesn't help that different people have contradictory theories, e.g. meat, eggs, and diary are either very important to eat, or very important to avoid. More precisely, the best form of meat is fish. Except you shouldn't eat fish, because they are full of deadly mercury. Villiam, Nutrition is Satisficing
I really liked this comment, so I wanted to hone in on it.
Suppose we have 3 options for a protein source for dinner: Chicken, steak, or fish. They're all considered to be pretty good sources of protein, by most doctors, but they all have downsides:
 Steak is linked to heart disease,
 fish contains higher levels of heavy metals, and
 chicken is just kind of bland.
What should you do?
A nutrition plan is a little bit like a stock portfolio, in that you can diversify away risk by investing a small amount in several different companies at once. However, many of the risks we talk about with nutrition are linked to the overconsumption of specific foods. That means that diversification is super effective!
So the smart satisficer's move is to have all three, but cycle them. If we're planning for the week, we might do chicken MondayWednesdayFriday, and then on the other 4 days alternate between steak and fish. This does require a bit of planning on your part, but if you stick to variations between just a handful of prettygood sources, I think it's pretty feasible.
(Note that I'm considering "Grilled chicken breast is really healthy, but I had it every night for three weeks and I'm never touching it again" as a strong long term and short term risk of overconsumption. In fact, since this seems to happen often with the healthiest foods we eat, I would strongly advise you to update on this prior as well: Extremely healthy foods need to be cycled in and out with their only very healthy, but tasty, options.)
Discuss
Intertheoretic utility comparison: examples
.mjxchtml {display: inlineblock; lineheight: 0; textindent: 0; textalign: left; texttransform: none; fontstyle: normal; fontweight: normal; fontsize: 100%; fontsizeadjust: none; letterspacing: normal; wordwrap: normal; wordspacing: normal; whitespace: nowrap; float: none; direction: ltr; maxwidth: none; maxheight: none; minwidth: 0; minheight: 0; border: 0; margin: 0; padding: 1px 0} .MJXcdisplay {display: block; textalign: center; margin: 1em 0; padding: 0} .mjxchtml[tabindex]:focus, body :focus .mjxchtml[tabindex] {display: inlinetable} .mjxfullwidth {textalign: center; display: tablecell!important; width: 10000em} .mjxmath {display: inlineblock; bordercollapse: separate; borderspacing: 0} .mjxmath * {display: inlineblock; webkitboxsizing: contentbox!important; mozboxsizing: contentbox!important; boxsizing: contentbox!important; textalign: left} .mjxnumerator {display: block; textalign: center} .mjxdenominator {display: block; textalign: center} .MJXcstacked {height: 0; position: relative} .MJXcstacked > * {position: absolute} .MJXcbevelled > * {display: inlineblock} .mjxstack {display: inlineblock} .mjxop {display: block} .mjxunder {display: tablecell} .mjxover {display: block} .mjxover > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxunder > * {paddingleft: 0px!important; paddingright: 0px!important} .mjxstack > .mjxsup {display: block} .mjxstack > .mjxsub {display: block} .mjxprestack > .mjxpresup {display: block} .mjxprestack > .mjxpresub {display: block} .mjxdelimh > .mjxchar {display: inlineblock} .mjxsurd {verticalalign: top} .mjxmphantom * {visibility: hidden} .mjxmerror {backgroundcolor: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; fontstyle: normal; fontsize: 90%} .mjxannotationxml {lineheight: normal} .mjxmenclose > svg {fill: none; stroke: currentColor} .mjxmtr {display: tablerow} .mjxmlabeledtr {display: tablerow} .mjxmtd {display: tablecell; textalign: center} .mjxlabel {display: tablerow} .mjxbox {display: inlineblock} .mjxblock {display: block} .mjxspan {display: inline} .mjxchar {display: block; whitespace: pre} .mjxitable {display: inlinetable; width: auto} .mjxrow {display: tablerow} .mjxcell {display: tablecell} .mjxtable {display: table; width: 100%} .mjxline {display: block; height: 0} .mjxstrut {width: 0; paddingtop: 1em} .mjxvsize {width: 0} .MJXcspace1 {marginleft: .167em} .MJXcspace2 {marginleft: .222em} .MJXcspace3 {marginleft: .278em} .mjxtest.mjxtestdisplay {display: table!important} .mjxtest.mjxtestinline {display: inline!important; marginright: 1px} .mjxtest.mjxtestdefault {display: block!important; clear: both} .mjxexbox {display: inlineblock!important; position: absolute; overflow: hidden; minheight: 0; maxheight: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjxtestinline .mjxleftbox {display: inlineblock; width: 0; float: left} .mjxtestinline .mjxrightbox {display: inlineblock; width: 0; float: right} .mjxtestdisplay .mjxrightbox {display: tablecell!important; width: 10000em!important; minwidth: 0; maxwidth: none; padding: 0; border: 0; margin: 0} .MJXcTeXunknownR {fontfamily: monospace; fontstyle: normal; fontweight: normal} .MJXcTeXunknownI {fontfamily: monospace; fontstyle: italic; fontweight: normal} .MJXcTeXunknownB {fontfamily: monospace; fontstyle: normal; fontweight: bold} .MJXcTeXunknownBI {fontfamily: monospace; fontstyle: italic; fontweight: bold} .MJXcTeXamsR {fontfamily: MJXcTeXamsR,MJXcTeXamsRw} .MJXcTeXcalB {fontfamily: MJXcTeXcalB,MJXcTeXcalBx,MJXcTeXcalBw} .MJXcTeXfrakR {fontfamily: MJXcTeXfrakR,MJXcTeXfrakRw} .MJXcTeXfrakB {fontfamily: MJXcTeXfrakB,MJXcTeXfrakBx,MJXcTeXfrakBw} .MJXcTeXmathBI {fontfamily: MJXcTeXmathBI,MJXcTeXmathBIx,MJXcTeXmathBIw} .MJXcTeXsansR {fontfamily: MJXcTeXsansR,MJXcTeXsansRw} .MJXcTeXsansB {fontfamily: MJXcTeXsansB,MJXcTeXsansBx,MJXcTeXsansBw} .MJXcTeXsansI {fontfamily: MJXcTeXsansI,MJXcTeXsansIx,MJXcTeXsansIw} .MJXcTeXscriptR {fontfamily: MJXcTeXscriptR,MJXcTeXscriptRw} .MJXcTeXtypeR {fontfamily: MJXcTeXtypeR,MJXcTeXtypeRw} .MJXcTeXcalR {fontfamily: MJXcTeXcalR,MJXcTeXcalRw} .MJXcTeXmainB {fontfamily: MJXcTeXmainB,MJXcTeXmainBx,MJXcTeXmainBw} .MJXcTeXmainI {fontfamily: MJXcTeXmainI,MJXcTeXmainIx,MJXcTeXmainIw} .MJXcTeXmainR {fontfamily: MJXcTeXmainR,MJXcTeXmainRw} .MJXcTeXmathI {fontfamily: MJXcTeXmathI,MJXcTeXmathIx,MJXcTeXmathIw} .MJXcTeXsize1R {fontfamily: MJXcTeXsize1R,MJXcTeXsize1Rw} .MJXcTeXsize2R {fontfamily: MJXcTeXsize2R,MJXcTeXsize2Rw} .MJXcTeXsize3R {fontfamily: MJXcTeXsize3R,MJXcTeXsize3Rw} .MJXcTeXsize4R {fontfamily: MJXcTeXsize4R,MJXcTeXsize4Rw} .MJXcTeXvecR {fontfamily: MJXcTeXvecR,MJXcTeXvecRw} .MJXcTeXvecB {fontfamily: MJXcTeXvecB,MJXcTeXvecBx,MJXcTeXvecBw} @fontface {fontfamily: MJXcTeXamsR; src: local('MathJax_AMS'), local('MathJax_AMSRegular')} @fontface {fontfamily: MJXcTeXamsRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_AMSRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_AMSRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_AMSRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalB; src: local('MathJax_Caligraphic Bold'), local('MathJax_CaligraphicBold')} @fontface {fontfamily: MJXcTeXcalBx; src: local('MathJax_Caligraphic'); fontweight: bold} @fontface {fontfamily: MJXcTeXcalBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakR; src: local('MathJax_Fraktur'), local('MathJax_FrakturRegular')} @fontface {fontfamily: MJXcTeXfrakRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXfrakB; src: local('MathJax_Fraktur Bold'), local('MathJax_FrakturBold')} @fontface {fontfamily: MJXcTeXfrakBx; src: local('MathJax_Fraktur'); fontweight: bold} @fontface {fontfamily: MJXcTeXfrakBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_FrakturBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_FrakturBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_FrakturBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathBI; src: local('MathJax_Math BoldItalic'), local('MathJax_MathBoldItalic')} @fontface {fontfamily: MJXcTeXmathBIx; src: local('MathJax_Math'); fontweight: bold; fontstyle: italic} @fontface {fontfamily: MJXcTeXmathBIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathBoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathBoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathBoldItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansR; src: local('MathJax_SansSerif'), local('MathJax_SansSerifRegular')} @fontface {fontfamily: MJXcTeXsansRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansB; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerifBold')} @fontface {fontfamily: MJXcTeXsansBx; src: local('MathJax_SansSerif'); fontweight: bold} @fontface {fontfamily: MJXcTeXsansBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsansI; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerifItalic')} @fontface {fontfamily: MJXcTeXsansIx; src: local('MathJax_SansSerif'); fontstyle: italic} @fontface {fontfamily: MJXcTeXsansIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_SansSerifItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_SansSerifItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_SansSerifItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXscriptR; src: local('MathJax_Script'), local('MathJax_ScriptRegular')} @fontface {fontfamily: MJXcTeXscriptRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_ScriptRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_ScriptRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_ScriptRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXtypeR; src: local('MathJax_Typewriter'), local('MathJax_TypewriterRegular')} @fontface {fontfamily: MJXcTeXtypeRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_TypewriterRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_TypewriterRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_TypewriterRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXcalR; src: local('MathJax_Caligraphic'), local('MathJax_CaligraphicRegular')} @fontface {fontfamily: MJXcTeXcalRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_CaligraphicRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_CaligraphicRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_CaligraphicRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainB; src: local('MathJax_Main Bold'), local('MathJax_MainBold')} @fontface {fontfamily: MJXcTeXmainBx; src: local('MathJax_Main'); fontweight: bold} @fontface {fontfamily: MJXcTeXmainBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainBold.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainI; src: local('MathJax_Main Italic'), local('MathJax_MainItalic')} @fontface {fontfamily: MJXcTeXmainIx; src: local('MathJax_Main'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmainIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmainR; src: local('MathJax_Main'), local('MathJax_MainRegular')} @fontface {fontfamily: MJXcTeXmainRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MainRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MainRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MainRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXmathI; src: local('MathJax_Math Italic'), local('MathJax_MathItalic')} @fontface {fontfamily: MJXcTeXmathIx; src: local('MathJax_Math'); fontstyle: italic} @fontface {fontfamily: MJXcTeXmathIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_MathItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_MathItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_MathItalic.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize1R; src: local('MathJax_Size1'), local('MathJax_Size1Regular')} @fontface {fontfamily: MJXcTeXsize1Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size1Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size1Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size1Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize2R; src: local('MathJax_Size2'), local('MathJax_Size2Regular')} @fontface {fontfamily: MJXcTeXsize2Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size2Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size2Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size2Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize3R; src: local('MathJax_Size3'), local('MathJax_Size3Regular')} @fontface {fontfamily: MJXcTeXsize3Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size3Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size3Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size3Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXsize4R; src: local('MathJax_Size4'), local('MathJax_Size4Regular')} @fontface {fontfamily: MJXcTeXsize4Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_Size4Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_Size4Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_Size4Regular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecR; src: local('MathJax_Vector'), local('MathJax_VectorRegular')} @fontface {fontfamily: MJXcTeXvecRw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorRegular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorRegular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorRegular.otf') format('opentype')} @fontface {fontfamily: MJXcTeXvecB; src: local('MathJax_Vector Bold'), local('MathJax_VectorBold')} @fontface {fontfamily: MJXcTeXvecBx; src: local('MathJax_Vector'); fontweight: bold} @fontface {fontfamily: MJXcTeXvecBw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/eot/MathJax_VectorBold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/woff/MathJax_VectorBold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTMLCSS/TeX/otf/MathJax_VectorBold.otf') format('opentype')}
A previous post introduced the theory of intertheoretic utility comparison. This post will give examples of how to do that comparison, by normalising individual utility functions.
The methodsAll methods presented here obey the axioms of Relevant data, Continuity, Individual normalisation, and Symmetry. Later, we'll see which ones follow Utility reflection, Cloning indifference, Weak irrelevance, and Strong irrelevance.
Max, min, meanThe maximum of a utility function u is maxs∈Su(s), while the minimum is mins∈Su(s). The mean of u ∑s∈Su(s)/S.

The maxmin normalisation of [u] is the u∈[u] such that the maximum of u is 1 and the minimum is 0.

The maxmean normalisation of [u] is the u∈[u] such that the maximum of u is 1 and the mean is 0.
The maxmean normalisation has an interesting feature: it's precisely the amount of utility that an agent completely ignorant of its own utility, would pay to discover that utility (as a otherwise the agent would employ a random, 'mean', strategy).
For completeness, there is also:
 The meanmin normalisation of [u] is the u∈[u] such that the mean of u is 1 and the minimum is 0.
The last two methods find ways of controlling the spread of possible utilities. For any utility u, define the mean difference: ∑s,s′∈Su(s)−u(s′). And define the variance: ∑s∈S(u(s)−μ)2, where μ is the mean defined previously.
These lead naturally to:

The mean difference normalisation of [u] is the u∈[u] such that u has a mean difference of 1.

The variance normalisation of [u] is the u∈[u] such that u has a variance of 1.
The different normalisation methods obey the following axioms:
Property Maxmin Maxmean Meanmin Mean difference Variance Utility reflection YES NO NO YES YES Cloning indifference YES NO NO NO NO Weak Irrelevance YES YES YES NO YES Strong Irrelevance YES YES YES NO NOAs can be seen, maxmin normalisation, despite its crudeness, is the only one that obeys all the properties. If we have a measure on S, then ignoring the cloning axiom becomes more reasonable. Strong irrelevance can in fact be seen as an antivariance; it's because of its second order aspect that it fails this.
Discuss
RAISE AI Safety prerequisites map entirely in one post
All RAISE projects have been discontinued, and a postmortem is in our plans. One of those projects was the AI Safety prerequisites online course originally announced here. We're sharing the curriculum here in almost plaintext format so that people can easily find and access it. There is also a GDoc here. (Note: two other products of RAISE are series of lessons on IRL and IDA. They don't have such neatly formed curriculums, and they are still accessible as online lessons.)
It was supposed to be a guide for helping people who want to get into AI safety research. It contains only foundations of math topics (Logic and proof, ZF(C) Set theory, Computability theory to be precise), which are more useful for agent foundations stuff and not useful for machine learning stuff. It was planned to be extended to cover more topics, but that never happened.
How to use thisThe main path contains 20 levels. It is the recommended path through the curriculum. Its levels are actually short sequences of levels from the three other paths.
To see what textbooks are required to study a path, see its beginning. Computability theory and set theory paths require two paid textbooks.
13 out of 20 levels of the main path are covered by our online course, which is free (but still requires paid textbooks). To use the online course, register here. You might prefer to use it instead of the text below because it provides more sense of progress, contains solutions to the exercises, has some writing mistakes fixed, maybe feels like a less tedious thing, and provides some additional exercises which we don't think are important.
CreditsThe curriculum was made by Erik Istre and Trent Fowler. People who created the online course are: Philip Blagoveschensky, Davide Zagami, Toon Alfrink, Hoagy Cunningham, Danilo Naiff, Lewis Hammond. Also these people contributed: Jacob Spence, Roland Pihlakas. Also, thanks to Grasple for providing their services for free.
Main path Level 1. Basic logicThe big ideas:
 Sentential Logic
 Truth Tables
 Predicate Logic
 Methods of Mathematical Proof
To move to the next level you need to be able to:
 Translate informal arguments into formal logic.
 Evaluate an argument as either valid or invalid.
 Explain how to prove an implication/conditional, a conjunction, a disjunction, and a negation and know what this looks like informally (i.e. in words and not symbols).
Why this is important:
 This builds the basic knowledge you need to be able to produce and understand mathematical proof. A firm foundation in how logical machinery operates is the best way to be assured that a proof you produce or read is correct. This also teaches the basic methods by which a proof is produced.
Skill Guides for this Level:
 Logic and proof path. Level 1. Basic logic
 Logic and proof path. Level 2. Quantified logic. Introduction to mathematical arguments
The big ideas:
 Axioms of Set Theory
 Set Operations
To move to the next level you need to be able to:
 Explain what a set is.
 Calculate the intersection, union and difference of sets.
 Prove two sets are equal.
 Apply basic axioms of ZermeloFraenkel set theory.
Why this is important:
 Set theory has become entrenched as the basic language with which all mathematics can be discussed. While there are more estranged parts of set theory that will likely be irrelevant to you, a fluency in the basic materials of set theory is necessary to understand more advanced mathematics.
Skill Guides for this level:
Level 3. Set Theoretic Relations and EnumerabilityThe big ideas:
 Ordered Pairs
 Relations
 Functions
 Enumerability
 Diagonalization
To move to the next level you need to be able to:
 Define functions in terms of relations, relations in terms of ordered pairs, and ordered pairs in terms of sets.
 Define what a onetoone (or injective) and onto (or surjective) function is. A function that is both is called a onetoone correspondence (or bijective).
 Prove a function is onetoone and/or onto.
 Explain the difference between an enumerable and a nonenumerable set.
Why this is important:
 Establishing that a function is onetoone and/or onto will be important in a myriad of circumstances, including proofs that two sets are of the same size, and is needed in establishing (most) isomorphisms.
 Equivalence relations and partial orderings are essential mathematical concepts which are powerful tools that can be used to analyze other mathematical objects or build new ones.
 Enumerability and nonenumerability introduces the difference between something being computable and noncomputable.
Skill Guides for this level:
 Computability theory path. Level 1. Enumerability and diagonalization
 Set theory path. Level 2. Set theoretic relations
The big ideas:
 Formal Semantics
 Model
To move to the next level you need to be able to:
 Evaluate the truth value of logical sentences in a given model.
 Build models for a set of logical sentences and then use those models to deduce information about the sentences.
Why this is important:
 An AI built on a formal system will reason based on some sort of proof and model theory. The former gives its methods of proof (which you’ll learn in the next chapter), and the latter its semantics.
 This level will give you your first sense of what models of logical sentences look like.
Skill guides for this level:
Level 5. Formal ProofThe big ideas:
 Natural Deduction Proof System
To move to the next level you need to be able to:
 Explain the difference between a formal system of proof and our informal notion of proof.
 Derive proofs of logical formula in the system of natural deduction.
Why this is important:
 Learning mathematical proof is hard, as you may have experienced in levels 2 and 3. By learning a formal system of proof you will make your own thoughts more rigorous, understand the smallest details that need to be covered to perform a proof, and build your intuition for informal proof.
Skill guides for this level:
Milestone  You’re now a !
 You now understand the basics of logic and how they apply to proof.
 You can take an argument and peel away the content to look purely at its structural details.
 You understand the basics of building a model of formal sentences of logic.
 You know the basic building blocks of set theory.
 You now know there exist uncomputable functions, and you know how to rigorously define the concept of a function.
Take a moment to reflect on what you’ve learned so far. Remind yourself of the important concepts that you’ve come across, and mark down any concepts which are still giving you trouble. Determine whether you need to seek out further resources to clear up any confusions.
Level 6. Turing Machines and the Halting ProblemThe big ideas:
 Turing Machine
 The Halting Problem
To move to the next level you need to be able to:
 Describe a turing machine and write basic turing algorithms.
 Give the basic idea of the halting problem.
 Give the basic idea of the proof that the halting function isn’t computable.
Why this is important:
 This is the beginning of learning formal computability. This allows us to think about computation without the limits of a physical domain. The first step to reasoning about Artificial General Intelligence will be understanding how it can be done before we worry about its practical feasibility.
 The halting problem is a pervasive and annoying problem. It’ll come up again and again, in many disguises. If we could solve the halting problem, things would be easy. Since we can’t, things are hard. Understanding the limitations it imposes is important.
Skill guides for this level:
Level 7. Equivalence Relations and OrderingsThe big ideas:
 Equivalence Relations
 Partitions
 Orderings
To move to the next level you need to be able to:
 Explain the relationship between equivalence relations and partitions.
 Name two different kinds of orderings and the conditions on the ordering relations required for these kinds.
Why this is important:
 This level is about further building up your mathematical toolbox. While equivalence relations and orderings currently seem like random abstract notions, they are both very important. The former is gives you a way to instantiate different notions of equality, which is important in modeling. The latter appears in constructing models for another turing machine equivalent notion of computability, the lambda calculus.
Skill guides for this level:
Level 8. Abacus Computability and Mathematical Proof by InductionThe big ideas:
 Proof by Induction
 Abacus Computability
To move to the next level you need to be able to:
 Explain mathematical proof by induction.
 Explain the abacus machine and the differences between a turing machine and an abacus machine.
 Build basic algorithms for an abacus machine.
Why this is important:
 Mathematical proof by induction is a very powerful technique. While you’ll first learn to use it in the context of natural numbers, you’ll soon see it has applications beyond this domain.
 An abacus machine gives you a higher level means of expressing algorithms, allowing you to abstract away from the minute details of strokes on a tape.
Skill guides for this level:
 Logic and proof path. Level 5. Proof by induction
 Computability theory path. Level 3. Abacus computability
The big ideas:
 Representing Natural Numbers with Sets
To move to the next level you need to be able to:
 Explain the relationship between inductive sets and the set theoretic construction of the natural numbers.
 Use the method of mathematical induction to prove claims about the natural numbers.
Why this is important:
 This builds an understanding of how set theory is used to provide rigorous constructions of mathematical objects we usually take for granted, like the natural numbers. Learning how to provide these constructions is an important part of building models in model theory.
Skill guides for this level:
Level 10. Recursive FunctionsThe big ideas:
 Primitive Recursive Functions
 Recursive Functions
 Primitive Recursive Sets
 Recursive Sets
To move to the next level you need to be able to:
 Translate the formal syntax of the recursive functions into an informal representation of a function and vice versa.
 Explain the difference between primitive recursion and recursion.
 Explain the difference between semirecursive and recursive.
Why this is important:
 By this point in your FAI career, you may have developed some sort of fascination of the power of this idea of “recursion”. Well, now you’re learning that notion. It is powerful, but it can also get a bit complicated so take your time and on this concept.
Skill guides for this level:
Level 11. Set Theoretic RecursionThe big ideas:
 The Recursion Theorem
 Peano Axioms
To move to the next level you need to be able to:
 State the recursion theorem and explain how this theorem coincides with our informal notion of recursion.
 Use the recursion theorem to create new functions.
 Explain what the axioms of Peano arithmetic are.
Why this is important:
 This is recursion...again! (In true recursive spirit.) However, it’s a bit different this time. This time we’re showing a way to look at recursion from a set theory perspective. We can take the recursive functions of computability as given and do some things with them, or we can use our set theoretic foundations to build the theory of recursion itself. Think of this as further justification and an assurance of what we’re doing.
 You’ll also get a look at what are known as “Peano’s axioms”. If we assume these axioms, or alternatively provide a set theoretic structure in which they hold, the claim is that these axioms can give us all our theorems of informal arithmetic in a formal system. Formalization of arithmetic will be important later for deriving limitative results about formal systems.
Skill guides for this level:
Level 12. The Equivalence of Different Notions of ComputabilityThe big ideas:
 Equivalence of Turing Machines and Recursive Computability
To move to the next level you need to be able to:
 Describe the general process of coding turing machines.
 Explain what a universal turing machine is.
Why this is important:
 Establishing the equivalence of independently developed methods of computation is a strong argument in favor of the ChurchTuring thesis.
 The existence of a universal turing machine is a good thing. It made the general purpose computer that we’re so used to today a feasible invention. We don’t need 1000 specialized machines, but only one that can imitate those 1000.
Skill guides for this level:
Milestone  You’re now a !
 You now understand the basic abstract representations of computation.
 You know some of what we can expect from computers and also what we can’t expect from them.
 You know a lot more set theoretic tools like equivalence relations and orderings.
 You’ve seen the construction of the natural numbers from the perspective of set theory.
 You know about mathematical induction and you’ve used it!
 You know about recursion and you’ve used it!
 More abstractly, your mathematical maturity has greatly increased from when you began.
As last time, take a moment to reflect on what you’ve learned so far. Remind yourself of the important concepts that you’ve come across, and mark down any concepts which are still giving you trouble. Determine whether you need to seek out further resources to clear up any confusions.
Level 13. IsomorphismsThe big ideas:
 Isomorphism
To move to the next level you need to be able to:
 Define isomorphism between structures.
 Prove that two structures are isomorphic.
Why this is important:
 Isomorphism is one of the most fundamental mathematical concepts. Finding isomorphisms allows you to move insights from one problem domain into another, which can be incredibly useful.
 However, it should be noted that “isomorphism” only rigorously applies when talking about algebraic structures, i.e. structures with relations and operations and a domain.
Skill guides for this level:
Level 14. Logic Review and The Relationship Between Computation and LogicThe big ideas:
 First Order Logic Syntax
 First Order Logic Semantics
 The Relationship Between Logic and Computability
To move to the next level you need to be able to:
 Read and translate the syntax for formal logic, and translate informal sentences to logic.
 Build models to evaluate logical claims.
 Explain how logic and turing machines/primitive recursion are related and how this leads to showing the undecidability of firstorder logic.
Why this is important:
 This chapter reviews concepts from Levels 4 and 5 so it’s important for the same reasons covered there.
 The undecidability of logic has important implications for how an Artificial General Intelligence is limited (in much the same way we are!) in deriving proofs.
Skill guides for this level:
 Computability theory path. Level 6. First order logic
 Computability theory path. Level 7. Undecidability of first order logic
The big ideas:
 Finite Set
 Countable Set
To move to the next level you need to be able to:
 State the CantorBernstein Theorem.
 Understand basic properties of finite and countable sets.
 Define what a countable set is.
 Prove that a set is of countable size.
Why this is important:
 The CantorBernstein theorem may not have strict relevance for you. You can consider it part of your general mathematical wellbeing. It’s some set theory you really should know.
 Finite and countable sizes are what we as finite beings have “constructive” access to. We reason about other cardinalities, but it all becomes a little less graspable and tangible. These sizes of sets are important and probably give us everything we need in the real world. (For more on this: Computable Real Numbers)
Skill guides for this level:
Level 16 (elective). Linear Orders and Completing the Real NumbersThe big ideas:
 Constructing the Real Numbers
To move to the next level you need to be able to:
 Define when two linear orders are similar.
 Explain how the concept of completeness motivates the real numbers.
 Explain how Dedekind cuts generate completeness for the rational numbers.
Why this is important:
 If we look at the rationals from the perspective of completeness, we motivate the development and construction of the real numbers. This provides another useful example of a mathematical construction.
 In the current context, the real numbers are a prime example of an uncountable set.
Skill guides for this level:
Level 17. Basic Model TheoryThe big ideas:
 Models
 Soundness
 Completeness
 Sequent Calculus
To move to the next level you need to be able to:
 Define a model.
 Explain the LowenheimSkolem theorem and the compactness theorem.
 Use a Gentzen style system or sequent calculus to derive proofs.
 Explain the concepts of soundness and completeness.
Why this is important:
 Model theory allows us to map a formal proof theoretic system into a defined domain where it hopefully applies. Whether or not it applies appropriately depends on the soundness and completeness results. This is all about asking “Do my formal deductions match up with something out there and do all of the somethings out there match up with my formal deductions?” As you may already know or will see in a few levels, the latter question is impossible to prove in formalized situations, but still worth pondering.
Skill guides for this level:
Milestone  You’re now a !
 You know the rigorous definition of an isomorphism.
 You now can explain the relationship between computability and logic.
 You understand way more about the size of things and way more sophisticated ways to count than you ever thought necessary.
 You can give a general outline of how to build the real numbers from the rationals.
 You can use a Gentzen style system to develop proofs.
 You can use soundness and completeness to demonstrate those Gentzen style proofs are “nice”.
 You know how to think about models more indepth and about some of the more important theorems that apply to them.
As last time, take a moment to reflect on what you’ve learned so far. Remind yourself of the important concepts that you’ve come across, and mark down any concepts which are still giving you trouble. Determine whether you need to seek out further resources to clear up any confusions.
Level 18 (elective). A quick look at cardinal and ordinal numbersTo move to the next level you need to be able to:
 Explain what a cardinal number is.
 Demonstrate the general idea behind the proof of Cantor’s Theorem.
 Explain what an ordinal number is.
 Use the generalized notions of transfinite induction and transfinite recursion to prove statements about large sets.
Why this is important:
 This level is mainly for expanding what set theoretic tools you’re familiar with. While it is unlikely that cardinal theory or ordinal theory will directly play into AGI research, it still may crop up in unexpected places like model theory.
 Ordinal theory also sets you up to later learn what is known as “proof theoretic strength” of a formal system which is expressed in ordinal numbers and relies on transfinite induction.
Skill guides for this level:
 Set theory path. Elective: Level 9. Cardinal numbers
 Set theory path. Elective: Level 10. Ordinal numbers. Axiom of replacement. Transfinite induction and recursion
The big ideas:
 The Axiom of Choice
 Arithmetization of Syntax
 Mathematical Induction
 Representability of Functions
To move to the next level you need to be able to:
 Elective: State the axiom of choice and at least loosely explain it to someone with basic set theory.
 Describe the process of generating Godel numbers for logical statements.
 Define what it means for a recursive function to be representable in a system of arithmetic.
Why this is important:
 Being aware of the axiom of choice makes it easier to be aware when you are invoking it. This generally points out how the apparently simplest of assumptions can turn out to be just that: assumptions. We must be prepared to discover all of our assumptions working into our informal proofs before we think about applying them to a potentially hazardous problem like an AGI.
 The rest of this level is setting up the material for the oftquoted Godel incompleteness theorems which place hard limits on what formal systems are capable of.
Skill guides for this level:
 Set theory path. Elective: Level 11. Axiom of choice
 Computability theory path. Level 9. Arithmetization. Representability of recursive functions
The big ideas:
 Godel’s Incompleteness Theorems
 Independence Result
 ZermeloFraenkel Set Theory
To move to the next level you need to be able to:
 Explain the diagonal lemma and why it leads to the limitative results of formal systems. Further, explain under what precise conditions we must be for these results to accurately apply.
 Explain Godel’s first and second incompleteness theorems and the general procedure of their proofs.
 Define what an independence result is and a particular example of such a result for ZFC.
 Explain why Godel’s Incompleteness theorems also apply to a formal set theory like ZFC.
Why this is important:
 This level is all about formal systems and what they can’t do. These are the problems that will need to be circumvented, or at least shown to be irrelevant to a development of a formal AGI.
Skill guides for this level:
 Set theory path. Level 12. ZF(C) set theory
 Computability theory path. Level 10. Indefinability, undecidability, incompleteness. The unprovability of consistency.
Milestone  You’re now a !
 You’re now a wellrounded set theorist.
 You can explain how to code up logical statements and give them Godel Numbers.
 You can explain why this leads to limitative results in formal systems and what these results mean.
As last time, take a moment to reflect on what you’ve learned so far. Remind yourself of the important concepts that you’ve come across, and mark down any concepts which are still giving you trouble. Determine whether you need to seek out further resources to clear up any confusions.
Logic and proof pathThe main textbook used in this path is a free textbook "Forall x" version 1.4. download v1.4 another link for v1.4 latext version Note that numbering of exercises sometimes changes between versions.
Another resource used in this path is Introduction to Mathematical Arguments by Michael Hutchings.
Level 1. Basic logic
Read Chapter 1 of forall x, “What is Logic?”.
 Work the following exercises:
 Part A
 Part B
 Part C
 Part D
 Work the following exercises:

Read Chapter 2 of forall x, “Sentential Logic”
 Work the following exercises:
 Part C
 Part D
 Part G
 Part H
 Work the following exercises:

Read Chapter 3 of forall x, “Truth Tables”
 Work the following exercises:
 From Part A:
 1
 4
 5
 11
 From Part B:
 1
 2
 6
 8
 9
 10
 From Part C:
 2
 4
 From Part D:
 3
 6
 7
 8
 10
 Part E
 Extra Credit: Part F
 From Part A:
 Work the following exercises:
 Read Chapter 4 of forall x, “Quantified Logic”
 Less Important Sections: “Definite Descriptions”
 Part A
 From Part B:
 Work at least 5, one with the word “all”, one with the word “some”, and one with the word “no”.
 From Part D:
 1
 5
 6
 Part H
 Part I
 Part K
 All odd numbers.
 Read Introduction to Mathematical Arguments until the end of section 3
 Print/save/copy the table on page 9 and always have it handy
 Section 3.3 talks about groups, which you may have never heard of. Pay attention to the structure of the proof, since that is what we’re interested in here, and ignore the content if it seems confusing. Proof by uniqueness proceeds by saying “if I want to prove there is only one something, I assume I have two somethings and show that they turn out to be the same thing”.
 Work exercises:
 1
 2
 3
 Extra Credit: 4
 Read Chapter 5 of forall x, “Formal Semantics”
 Work Exercises:
 Part B
 Part D
 Part F: 1, 2, 3, 5, 8
 Part G: 3, 6, 10
 Part H: 1, 4, 5, 9
 Part I: 1, 2, 3
 Extra Credit: Part J
 Work Exercises:
 Read Chapter 6 of forall x, “Proofs”
 Part A
 Part B
 Part E: 1, 2, 5
 Part G: 1, 3, 4
 Part I
 Part J
 Part Q
 Part T: 1, 2, 4
 Part U: 1, 2, 3, 7, 8, 10
 Extra Credit: Part M
 Read Section 4 of Introduction to Mathematical Arguments , “Proof by Induction”
 This is a very powerful and important proof technique to get comfortable with. Make sure you understand the structure of this argument.
 Work the following exercises:
 1
 2
 4
 Extra Credit:
 4
 Optional Read Appendix “Sets”
 This will provide a review of the basic concepts of sets if you want a refresher/reinforcement.
 All of the exercises for the appendix are good checks on your knowledge of set theory.
 For 8, draw pictures to guess at new relationships and then prove them or disprove them by finding a counterexample.
This path uses the Introduction to set theory. Third edition. Revised and expanded by Hrbacek, Jech.
Level 1. Basic set theory In Introduction to Set Theory by Hrbacek and Jech
 Read Chapter 1 Section 1 “Introduction to Sets”
 Read Chapter 1 Section 2 “Properties”
 Read Chapter 1 Section 3 “The Axioms”

Work the following exercises:
 3.1
 3.2
 This question is equivalent to asking you to prove:
 “If the Weak Axiom of Existence and the Comprehension Schema hold, then the empty set exists.”
 This is a general note to always try to translate a mathematical assertion into a straightforward logical statement. Once you have it in that logical statement, then you can rely on your logical knowledge to know general steps you need to do to prove that statement.
 This question is equivalent to asking you to prove:

3.4
 Proving this requires a few steps that are not immediately obvious.
 First, prove there is a set that contains A and B. (What axioms implies this?)
 Second, prove there is a set that contains the elements of A and B. (Again, what axiom?)
 Finally, use the Comprehension Schema on this set to get the desired set.
 Proving this requires a few steps that are not immediately obvious.

3.5a
 This one also requires a few steps as with 3.4. To find these steps, work backwards.
 What instance of the comprehension schema do you need to make this set exist?
 What other set’s existence does that instance of the comprehension schema rely on?
 How do you make that one exist?
 Does this new set rely on another one for existence?
 Continue to move backwards until you can be sure that you have a set that exists. You will have to rely on the fact that the existence of A, B, and C is assumed.
 This one also requires a few steps as with 3.4. To find these steps, work backwards.

Extra Credit: 3.3

 Read Chapter 1 Section 4
 Work the following exercises:
 4.1
 Do all formulas involving union, intersection, and difference.
 If you’re proving set equality between two arbitrary sets A and B, you are always starting with the assumption that you have an arbitrary element x in A and trying to show that it gets into B.
 Reread the axiom of extensionality and explain why you are trying to prove that “if x is in A then x is in B” and “if x is in B then x is in A”.
 Extra credit: The symmetric difference formulas.
 4.2a
 How do you prove a chain of “if and only if”? Investigate what it means to prove for arbitrary sentences A, B, and C: A iff B iff C.
 4.2b
 4.3
 Extra Credit: 4.4
 4.1
 Work the following exercises:
This level uses Introduction to Set Theory by Hrbacek and Jech.
 The level of abstraction will start getting a bit higher as we progress. We’ll be defining objects to define other objects to define other objects. Check yourself and make sure you know what each term refers to and what each terms means as the abstraction progresses.
 Read Chapter 2 Section 1 “Ordered Pairs”
 Work the following exercises:
 1.1
 Part of 1.2
 It’s enough to work out the proof of existence for (a, b). This is to review your axioms.
 1.3
 Extra Credit: 1.6
 This is mainly interesting to state the equivalent theorem. This illustrates that set theory can represent the same concept in multiple ways. The proof of this equivalent theorem is tedious and unenlightening after going through Theorem 1.2.
 Work the following exercises:
 Read Chapter 2 Section 2 “Relations”
 Work the following exercises:
 2.3 Parts: a, b, c, d
 2.4 Parts: a, d
 2.6
 Extra Credit: 2.1
 Further axiom review.
 Work the following exercises:
 Read Chapter 2 Section 3 “Functions”
 Pay special attention to:
 Definition 3.1
 Definition 3.3
 Definition 3.7
 A function that is onetoone is also called injective, and a function that is onto is also called surjective. A function that is both is called a onetoone correspondence or a bijection.
 Pay special attention to:
 Work the following exercises:
 3.1
 3.2
 3.3
 3.4a
 3.5
 3.6
 Extra credit:
 3.10
 3.11
Resources: Introduction to Set Theory by Hrbacek and Jech
 Read Chapter 2 Section 4 “Equivalences and Partitions”
 Pay special attention to:
 Definition 4.1
 Definition 4.3
 Definition 4.6
 Theorem 4.10
 Work the following exercises:
 Prove Theorem 4.10.
 4.1
 Not necessary to produce proofs for this one, play with the relations and see what you come up with.
 4.2
 Pay special attention to:
 Read Chapter 2 Section 5 “Orderings”
 Pay special attention to:
 Definition 5.2
 Definition 5.5
 Work the following exercises:
 5.1
 5.3
 5.5
 5.7
 5.12
 Extra Credit:
 5.13
 5.14
 Pay special attention to:
Resources: Introduction to Set Theory by Hrbacek and Jech

Read Chapter 3 “Natural Numbers” Section 1 “Introduction to Natural Numbers”
 Work the following exercises:
 1.1
 For the second half of 1.1, assume there is such a z and reason from there. Refer to Introduction to Mathematical Arguments Proof by Contradiction and/or Uniqueness proofs.
 1.1
 Work the following exercises:

Read Chapter 3 Section 2 “Properties of Natural Numbers”
 Pay special attention to:
 The Induction Principle
 Work the following exercises:
 2.1
 2.2
 2.3
 2.4
 2.6
 2.7
 Extra Credit:
 2.11
 2.12
 2.13
 Pay special attention to:

Read Chapter 3 Section 3 “The Recursion Theorem”
 Pay special attention to:
 The Recursion Theorem
 Work the following exercises:
 3.1
 3.2
 Extra Credit:
 3.5
 3.6
 Pay special attention to:

Read Chapter 3 Section 4 “Arithmetic of Natural Numbers”
 Pay special attention to:
 The Peano Axioms
 Work the following exercises:
 4.1
 4.2
 4.3
 4.4
 4.5
 Extra Credit:
 4.7
 4.8
 Pay special attention to:
Resources: Introduction to Set Theory by Hrbacek and Jech
 Read Chapter 3 Section 3 “The Recursion Theorem”
 Pay special attention to:
 The Recursion Theorem
 Work the following exercises:
 3.1
 3.2
 Extra Credit:
 3.5
 3.6
 Pay special attention to:
 Read Chapter 3 Section 4 “Arithmetic of Natural Numbers”
 Pay special attention to:
 The Peano Axioms
 Work the following exercises:
 4.1
 4.2
 4.3
 4.4
 4.5
 Extra Credit:
 4.7
 4.8
 Pay special attention to:
Resources: Introduction to Set Theory by Hrbacek and Jech
 Read Chapter 3 Section 5 “Operations and Structures”
 Pay special attention to:
 Definition 5.6
 This is a more abstract notion of isomorphism. It is dependent not only on a bijection, but making sure that structural features are preserved (like relations or functions).
 Work the following exercises:
 5.1
 5.4
 5.6
 5.12
 Pay special attention to:
Resources: Introduction to Set Theory by Hrbacek and Jech
 Read Chapter 4 “Finite, Countable, and Uncountable Sets” Section 1 “Cardinality of Sets”
 Pay special attention to:
 Theorem 1.6 (CantorBernstein)
 Work the following exercises:
 1.1
 1.2
 1.5
 Extra Credit:
 1.10
 1.11
 1.12
 Pay special attention to:
 Read Chapter 4 Section 2 “Finite Sets”
 Pay special attention to:
 Lemma 2.2
 This lemma is also known as the pigeonhole principle in combinatorics. In fact, most of this chapter is teaching you the basic counting methods of finite combinatorics, which is important in probabilities of events with finitely many outcomes.
 Lemma 2.2
 Work the following exercises:
 2.1
 2.2
 2.3
 2.5
 Pay special attention to:
 Read Chapter 4 Section 3 “Countable Sets”
 Pay special attention to:
 Corollary 3.6
 This gives you a vague idea of how “small” countable sets are compared to uncountable.
 Corollary 3.6
 Work the following exercises:
 3.1
 3.2
 3.3
 Extra Credit:
 3.5
 3.6
 Pay special attention to:
Resources: Introduction to Set Theory by Hrbacek and Jech
 Read Chapter 4 Section 4 “Linear Orderings”
 Work the following exercises:
 4.1
 4.3
 Work the following exercises:
 Read Chapter 4 Section 5 “Complete Linear Orderings”
 Pay special attention to:
 Theorem 1.3
 Definition 5.5
 Definition 5.6
 Work the following exercises:
 5.1
 For a little review of proof by contradiction.
 5.2
 Extra credit:
 5.5
 5.8
 5.1
 Pay special attention to:
 Read Chapter 4 Section 6 “Uncountable Sets”
 Pay special attention to:
 Cantor’s Proof of Theorem 6.1
 Work the following exercises:
 6.1
 Pay special attention to:
Resources: Introduction to Set Theory by Hrbacek and Jech
 Don’t get stuck in this level. Become familiar with these ideas and then move on. If you’re working in computation, countable sets are mostly what you worry about. It’s nice to be aware of these properties of cardinality to be more set theoretically wellrounded.
 Read Chapter 5 “Cardinal Numbers” Section 1 “Cardinal Arithmetic”
 Pay special attention to:
 Theorem 1.8 (Cantor’s Theorem)
 Theorem 1.9
 Don’t spend too much time on:
 The operations of cardinal arithmetic. Unlikely to be very important to you.
 Work the following exercises:
 1.5
 1.7
 Pay special attention to:
 Read Chapter 5 “Cardinal Numbers” Section 2 “The Cardinality of the Continuum”
Resources: Introduction to Set Theory by Hrbacek and Jech
 Same as the previous level. Ordinal numbers won’t be terribly important except in some model theoretic cases. Don’t get bogged down, this is probably way more ordinal theory than you’ll need. Become familiar with the ideas, and then move on.
 Read Chapter 6 “Ordinal Numbers” Section 1 “WellOrdered Sets”
 Work the following exercises:
 1.4
 Work the following exercises:
 Read Chapter 6 Section 2 “Ordinal Numbers”
 Pay special attention to:
 Definition 2.2
 Work the following exercises:
 2.1
 2.2
 2.3
 Pay special attention to:
 Read Chapter 6 Section 3 “The Axiom of Replacement”
 Work the following exercises:
 3.1
 Work the following exercises:
 Read Chapter 6 Section 4 “Transfinite Induction and Recursion”
 Pay special attention to:
 4.1 Transfinite Induction
 4.5 Transfinite Recursion
 Pay special attention to:
 Neither section 5 or section 6 is likely to be useful to you. Read if you’re interested but don’t feel compelled to.
Resources: Introduction to Set Theory by Hrbacek and Jech
 This level is for further mathematical enrichment. Anyone claiming to be skilled in set theory should be aware of the axiom of choice and it’s consequences. However, as with the last two levels, you only need familiarity. Don’t get stuck here. The next level will return to set theory that is more important to you.
 Read Chapter 8 “The Axiom of Choice” Section 1 “The Axiom of Choice and Its Equivalences”
 Optional Read Chapter 8 Section 2 “The Use of the Axiom of Choice in Mathematics”
Resources: Introduction to Set Theory by Hrbacek and Jech
 We are again back to something that matters to AGI research: formal systems.
 Read Chapter 15 “The Axiomatic Set Theory” Section 1 “The ZermeloFraenkel Set Theory With Choice”
 Read Chapter 15 Section 2 “Consistency and Independence”
 The existence of independence results is part of the heart of the problem in building an AGI. How do we make a formal system capable of selfextension so as to cover as many problem domains as possible when we know our formal systems always have blind spots?
 Read Chapter 15 Section 3 “The Universe of Set Theory”
 This last section should give you an idea of what work is still being done in set theory, and that the field is far from closed. More generally, it shows that formal systems are in constant need of being tuned and pushed to their limits if we are to keep improving them.
This path uses Computability and Logic (Fifth Edition) by Boolos, Burgess and Jeffrey. An instructor's manual is available, it has hints for many problems.
Level 1. Enumerability and diagonalizationResources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 Read Chapter 1 “Enumerability”
 Pay special attention to:
 Example 1.2
 Example 1.3
 Example 1.9
 Example 1.13
 Work the following exercises:
 Note that to demonstrate a set is enumerable is to describe a function (or list) that will eventually list any element of the set. This description must be precise enough that anyone who inspects your enumeration becomes convinced that every element will in fact be listed.
 1.1
 1.2
 1.3
 1.5
 Hint: Example 1.9
 Extra credit: 1.7
 Pay special attention to:
 Read Chapter 2 “Diagonalization”
 Diagonalization is a concept that will begin to pop up in a lot of places. In general, it sketches out the boundaries of what is logically possible. It’ll come up again, so start trying to wrap your head around it.
 Pay special attention to:
 Theorem 2.1 and how it is proved
 Work the following exercises:
 2.1
 Hint: Refer to the proof of Theorem 2.1
 Another hint: Refer back to your solution of exercise 1.5
 Extra credit: 2.13
 2.1
Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 Read Chapter 3 “Turing Machines”
 Work the following exercises:
 3.1
 3.2
 3.3
 Extra Credit:
 3.5
 Work the following exercises:
 Read Chapter 4 “Uncomputability” Section 1 “The Halting Problem”
 Optional Read Chapter 4 Section 2 “The Productivity Function”
 Work the following exercises:
 4.1
 4.2
 Extra Credit:
 4.5
 Work the following exercises:
Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 Read Chapter 5 “Abacus Computability”
 Work the following exercises:
 5.1
 5.2
 5.4
 Extra Credit:
 5.3
 5.5
Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 Read Chapter 6 “Recursive Functions”
 While reading this chapter and before the exercises:
 What is the difference between a primitive recursive function and a recursive function?
 Practice unpacking the formal recursion syntax with functions given in the text. Do the reverse; practice tracing the steps to write them in formal syntax.
 Work the following exercises:
 6.1
 6.3
 Extra Credit:
 6.2
 6.7
 While reading this chapter and before the exercises:
 Read Chapter 7 “Recursive Sets and Relations” Sections 1 and 2
 Optional Read Chapter 7 Section 3 “Further Examples”
 Work the following exercises:
 7.1
 7.3
 7.5
 Extra Credit
 7.9
 Work the following exercises:
Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 Read Chapter 8 “Equivalent Definitions of Computability”
 This chapter’s main importance is in understanding the process of proving equivalence. Go through the proofs a couple time to ensure you are understanding. You don’t need to be able to reproduce them, but be able to follow.
 Extra Credit Exercise:
 8.1
Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 This level works out to be a review of first order logic and semantics. This also introduces you to a very important method of using proof by induction in logic.
 Read Chapter 9 “A Precis of FirstOrder Logic: Syntax”
 Work the following problems:
 9.1
 9.2
 9.3
 9.4
 Extra Credit:
 9.6
 9.7
 9.8
 Work the following problems:
 Read Chapter 10 “A Precis of FirstOrder Logic: Semantics”
 Work the following problems:
 10.1
 10.2
 10.3
 10.4
 10.7
 Extra Credit:
 10.8
 10.12
 10.14
 Work the following problems:
Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 Read Chapter 11 “The Undecidability of FirstOrder Logic”
 Work the following exercises:
 11.1
 11.2
 Extra Credit:
 11.12
 11.13
 Work the following exercises:
Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 Read Chapter 12 “Models”
 Work the following exercises:
 12.1
 12.2
 12.8
 12.15
 Extra Credit:
 12.6
 12.9
 12.19
 Work the following exercises:
 Read Chapter 13 “The Existence of Models”
 Familiarity with the ideas in this chapter is encouraged, but it is nonessential to spend a significant amount of time here.
 Read Chapter 14 “Proofs and Completeness”
 Work the following exercises:
 14.1
 14.2
 14.3
 14.5
 Extra Credit: 14.9
 Work the following exercises:
Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 This level builds the machinery to construct the proofs to Godel’s Incompleteness Theorems.
 Read Chapter 15 “Arithmetization”
 Work the following exercises:
 15.1
 15.2
 15.5
 Extra Credit:
 15.9
 Work the following exercises:
 Read Chapter 16 “Representability of Recursive Functions”
 Section 16.4 is a must. It is worth knowing the comparison between the system presented in this book and what else you might see in the literature.
 Work the following exercises:
 16.1
 16.2
 16.3
 16.10
 16.11
 Extra Credit:
 16.8
 16.9
 16.21
Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey
 The most important thing in the chapters of this level is to take the proofs slowly and understand them bit by bit.
 Read Chapter 17 “Indefinability, Undecidability, Incompleteness”
 Work the following exercises:
 17.1
 Work the following exercises:
 Read Chapter 18 “The Unprovability of Consistency”
Discuss
Страницы
 1
 2
 3
 4
 5
 6
 7
 8
 следующая ›
 последняя »