Вы здесь

Сборщик RSS-лент

What does failure look like?

Новости LessWrong.com - 4 часа 43 минуты назад
Published on May 19, 2022 12:59 AM GMT

Not related to AI.

I'm quite worried at the moment that there's some way that I can fail, become depressed, reclusive, and not achieve goals that's subtle enough that I won't notice any mistakes I'm making. I'm quite confident I won't become an alcoholic and won't take hard drugs, but there might be so many other ways to fail that are slower and harder to spot.

 

What are these ways? How do I avoid them? 



Discuss

Should you kiss it better?

Новости LessWrong.com - 8 часов 44 минуты назад
Published on May 19, 2022 3:58 AM GMT

Whenever my toddler hurts himself, I offer to kiss it better. This works like magic - he immediately stops crying and carries on playing happily. Placebos are wonderful things.

Doing this makes me happier, it makes him happier, and also serves as a useful diagnostic tool - if he carries on crying I know he's actually hurt himself badly.

Teaching children that kisses can make things better is definitely lying to them, but this seems like a good place to make a principled exception. WDYT?



Discuss

A possible check against motivated reasoning using elicit.org

Новости LessWrong.com - 13 часов 15 минут назад
Published on May 18, 2022 8:52 PM GMT

Are you worried you may be engaging in motivated reasoning, rationalization ... or committing other reasoning fallacies?

I propose  the following epistemic check using Elicit.org's "reason from one claim to another" tool

Whenever you have a theory that A→B.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} , 
Take your  theory negating one side or the other (or the contrapositive of either negation), and feed it into this tool.

Feed it
A→¬B

 and/or 
¬A→B,

and see if any of the arguments it presents seem equally plausible to your arguments for A→B.

If they seem similarly plausive, believe your original arguments and conclusion less. 

Caveat: the tool is not working great yet, and often requires a few rounds of iteration, selecting the better arguments and tell.ing it "show me more like this", or feeding it some arguments.

When in Rome ... do or don't do as the Romans do?

Discuss

Feminism and Femininity

Новости LessWrong.com - 13 часов 50 минут назад
Published on May 18, 2022 10:53 PM GMT

Is it feminist to wear lipstick? There are three possible views on the position that feminist thought should take with respect to expressions of conventional femininity in patriarchal societies: 

  • Expressing femininity is actively good
  • Expressing femininity is actively bad
  • Women should do whatever makes them feel most comfortable
  • Femininity is a distraction - masculinity is the problem  

These different views express a combination of different views on the mechanism by which patriarchy operates, the most effective way to advance women's liberation, and the duties which women have to their gender. 

How does patriarchy work

There are three competing views of how patriarchy might work. Patriarchy could be a result of gender socialisation. This theory goes that the way in which gender is socialised holds women back from holding power, achieving status and having good life chances - or some combination of all three. In contemporary Western Europe this mostly manifests as relatively soft norms where, for instance, women are more harshly punished for expressing anger. Probably the most important substantiation of gender socialisation as preventing women from achieving power and status - although not necessarily good life outcomes - is the norm where women sacrifice their careers when raising children with a male partner. In less gender equal societies this can take on a much more extreme form, where the common pattern is the barring of women from the public sphere. This describes societies such as classical Athens or 19th century Pashtun farmers where women are excluded from commerce and politics - the things that give men power.

The competing view is that patriarchy is defined by a value system rather than a set of norms. In the former theory what was deemed to be high status things - the axis along which inequality should be measured - was taken as a given. It was participating in the public square. Under this other view however, patriarchy comes from the work women do and femininity per se being defined as less valuable than the work men do and masculinity per se. 

There’s a strong version and a weak version of this claim. The strong version says that it really would be fine if women stayed within the private sphere if that work was valued equally to the work in the public sphere. To make this concrete, and staying within the context of capitalist societies in Western Europe, we could pay women equal amounts to the average male salary for care work that women disproportionately provide, both as a way of concretely giving women more power and better lives, but also because money is a key way in which these societies express who has, and what roles, are high status. This could be extended to other forms of traditionally female work in the public sphere, such as nursing and cleaning. 

However, there’s also a weaker and subtly different form of this claim. This version doesn’t argue that women should keep to their traditional roles but that these roles should be higher status, it argues that women should keep to traditional versions of femininity and society should change to value these traits more highly. For instance, rather than women adopting the risk taking attitudes of men, society should change to value lower risk more highly. 

The final view is that it’s nothing women do that’s the problem - it’s male power and often specifically male violence. This is the logic behind the focus on male violence against women despite the fact that much are much more at risk from homicide. In this worldview, male violence against women isn’t simply a tragedy it’s a part of how the patriarchy controls women. Murder is only the most extreme form of this. Domestic violence and sexual violence are probably the next rungs down the ladder, followed by sexual harassment and right down to, for instance, interrupting women. There’s also a more specific phenomena of women who don’t follow gender norms in some respect being harmed by men because of this. A good example of this is instance Margret Thatcher being portrayed as a man on spitting image, the joke being that she was domineering like a man so lets portray her as a man to take the piss haha very funny. Outside of a contemporary Western European context, honour killings are an extreme substantiation of this. 

Should feminists wear lipstick?

These three theories of patriarchy have very different implications for whether feminists should wear lipstick - i.e the degree to which women should express their femininity and much more generally how we should structure policy. The first view - where women are socialised to not challenge men’s position in the public space - implies that we should aim for a breaking down of femininity and gender and more generally. Gender itself is the rope that binds women. To put this in more concrete terms, this would imply that, for instance, we should be pushing back hard against little girls wearing pink and going to dance classes. For a more policy relevant example we should be extremely heavily incentivising men to take paternity leave with the goal of bringing men as equally into the private sphere as women so that women can take their fair share of the public sphere. 

If one believes the second then most definitely feminists should wear lipstick but not only that, women not expressing femininity in the public sphere is as anti-feminist as the sexualization of female politicians is under the first theory. By not expressing femininity in the public sphere women enforce the notion that femininity is inferior to masculine traits and makes it harder for other women to follow them. The strong version of this theory takes this argument even further. By privileging the public sphere over the private we’re implicitly subscribing to male values. This would imply that we should be paying mothers and other caring roles that women disproportionately take on as we would pay any high status, important job. 

Finally, the third theory says that the question of whether feminists should wear lipstick missies the point at best and a worst is another way of putting the onus on women to change their behaviour around the fixed point of men. The extreme version of this position argues for curfews on men to allow women to be able to go out in public free from the spectre of male violence. Less extreme ones argue for a shift in policing priorities to focus more male violence against women, harsh penalties for sexual harassment and strong norms against things like men interrupting women (and hopefully not for men writing articles about feminism who haven’t even read anything by Judith Butler.) 

Politics is about action

It’s not enough to merely know which one of these theories best describes how patriarchy works - politics is about action and the best action to take is determined by the constraints you face and your normative judgements. 

I think one of the key issues in deciding what policy course to take is the degree to which gender - and I mean gender not sex - is biologically determined rather than socially constructed. If one believes that it’s likely that women will always have a much stronger urge to spend a substantial portion of their lives looking after their children then it becomes a much higher priority to raise the status and pay of child rearing and care work more generally. To emphasise the point I’ve made a number of times I believe that there are trade offs here. If one believes that gender socialisation is the key culprit then raising the status of women spending a large portion of their lives as caregivers reduces the incentive for women to enter the public sphere and the traditionally male role that that entails and so is actively harmful to women's liberation. 

A second key consideration is how much one believes that what we value now is good. If it genuinely is the case that, for instance, being very willing to take big risks is very socially valuable this should push one towards attempting to change what femininity entails rather than attempting to devalue risk taking. Conversely, one could believe that what we currently value - both intrinsically and in terms of valuable traits - is deeply harmful. In this case, in addition to it being good for gender equality, having society value roles currently associated with femininity will make all of society better off. For instance, working extremely long hours at the expense of your family might be worth it if it’s the only way to create the most innovative products. Alternatively, if people are mostly working very long hours to create better versions of flavoured sugar water, maybe it would be good if society valued spending time with one's family more highly.

The question of how patriarchy arise and perpetuate is of course an extraordinarily complex question that requires careful empirical work and it’s beyond the scope of this post to review that literature here. 

The other core question of politics is normative - what is the right thing to do. I think it’s this lens that justifies the view that women should simply do what makes them most comfortable. This post has been written from an implicitly feminist perspective. The core normative feature of this perspective is that individuals have a special responsibility to reduce gender inequality - in some sense women who don’t are traitors to their gender. In the language of moral philosophy everyone and potentially women in particular have a duty to fight against patriarchy and an action which has gender at its core is only permissible if it reduces gender inequality. The view that women should do what makes them most comfortable is fundamentally a liberal position. Individuals don’t have special responsibilities to their identity groups even if those identity groups are oppressed. It may very well be a praiseworthy thing to reduce gender inequality in the way one lives one's everyday life, but it is not blameworthy if one does not, especially if one is prioritising other morally important goals. Think the busy doctor who focuses on saving the lives of her patients and takes the path of least resistance with respect to other issues. 

Synthesis 

Throughout this post I’ve emphasised the need for tradeoffs. But ideally one would practise policies that improve the lives of women regardless of the theory of patriarchy that one subscribes to. This might push one towards policies aimed at reducing domestic violence in high income countries. In low income countries, where because the cost of living is so much lower one can often have vastly more impact with the same amount of money, this might imply improved access to financial services to allow women to be more independent of their male relatives. Politics is about action and effectiveness. This post will have been a success if it, very marginally, helps us take better action.



Discuss

What's up with the recent monkeypox cases?

Новости LessWrong.com - 14 часов 36 минут назад
Published on May 18, 2022 10:06 PM GMT

Is it a thing I should be allocating attention to?

Wikipedia tells me that human-to-human transmission of monkeypox is pretty rare/difficult. There has been some community spread recently at least in the UK, but that's been speculated to have been from sex. On the other hand, there's been a bunch of cases reported in Portugal and Spain recently as well. Is that just normal background rate being given more attention than usual by the news cycle? Or is this actually an unusually high number of cases? Most importantly, is the number of cases significant evidence of increased human-to-human transmission?



Discuss

We have achieved Noob Gains in AI

Новости LessWrong.com - 15 часов 18 минут назад
Published on May 18, 2022 8:56 PM GMT

TL;DR I explain why I think AI research has been slowing down, not speeding up, in the past few years.

How have your expectations for the future of AI research changed in the past three years? Based on recent posts in this forum, it seems that results in text generation, protein folding, image synthesis, and other fields have accomplished feats beyond what was thought possible. From a bird's eye view, it seems as though the breakneck pace of AI research is already accelerating exponentially, which would make the safe bet on AI timelines quite short.

This way of thinking misses the reality on the front lines of AI research. Innovation is stalling beyond just throwing more computation at the problem, and the forces that made scaling computation cheaper or more effective are slowing. The past three years of AI results have been dominated by wealthy companies throwing very large models at novel problems. While this expands the economic impact of AI, it does not accelerate AI development.

To figure out whether AI development is actually accelerating, we need to answer a few key questions:

  1. What has changed in AI in the past three years?
  2. Why has it changed, and what factors have allowed that change?
  3. How have those underlying factors changed in the past three years?

By answering these fundamental questions, we can get a better understanding of how we should expect AI research to develop over the near future. And maybe along the way, you'll learn something about lifting weights too. We shall see.

What has changed in AI research in the past three years?

Gigantic models have achieved spectacular results on a large variety of tasks.

How large is the variety of tasks? In terms of domain area, quite varied. Advances have been made in major hard science problems like protein synthesis, imaginative tasks like creating images from descriptions, and playing complex games like Starcraft.

How large is the variety of models used? While each model features many domain specific model components and training components, the core of each of these models is a giant transformer trained with a variant of gradient descent, usually ADAM.

How large are these models? That depends. DALLE2 and AlphaFold are O(10GB), AlphaStar is O(1GB), and the current state of the art few shot NLP models (Chinchilla) are O(100GB).

One of the most consistent findings of the past decade of AI research is that larger models trained with more data get better results, especially transformers. If all of these models are built on top of the same underlying architecture, why is there so much variation in size?

Think of training models like lifting weights. What limits your ability to lift heavy weights?

  • Data availability: (Nutrition) If you don't eat enough food, you'll never gain muscle! Data is the food that makes models learn, and the more "muscle" you want the more "food" you need. When looking for text on the internet, it is easy to get terabytes of data to train a model. This is harder for other tasks
  • Cost (exhaustion): No matter how rich your corporation is, training a model is expensive. Each polished model you see comes after a lot of experimentation and trials, which uses a lot of computational resources. AI labs are notorious cost sinks. The talent they acquire is expensive, and in addition to their salaries the talent demands access to top of the line computational resources.
  • Training methodology (What exercises you do). NLP models only require to train one big transformer. More complex models like DALLE-2 and AlphaFold have many subcomponents optimized for their use cases. Training an NLP model is like deadlifting a loaded barbell and training AlphaFold is like lifting a box filled with stuff: at equivalent weight, the barbell is much easier to pick up because the load is balanced, uniform, and in one motion. When picking up the box, the weight is unevenly distributed which makes the task harder. Alphastar was trained by creating a league of AlphaStars which competed against each other in actual games. To continue our weightlifting analogy, this is like a higher rep range with lower weight.

Looked at this way, what has changed over the past three years? In short, we have discovered how to adapt a training method/exercise (the transformer) to a variety of use cases. This exercise allows us to engage our big muscles (scalable hardware and software optimized for transformers). Sure, some of these applications are more efficient than others, but overall they are way more efficient than what they were competing against. We have used this change in paradigm to "lift more weight", increasing the size and training cost of our model to achieve more impressive results.

(Think about how AlphaFold2 and Dalle-2, despite mostly being larger versions of their predecessors, drew more attention than their predecessors ever did. The prior work in the field paved the way by figuring out how to use transformers to solve these problems, and the attention comes from when they scaled the solution to achieve eye popping results. In our weightlifting analogy, we are learning a variation of an exercise. The hard part is learning the form that allows you to leverage the same muscles, but the impressive looking part is adding a lot of weight.)

Why and how has it changed?

In other words: why are we only training gigantic models and getting impressive results now?

There are many reasons for this, but the most important one is that no one had the infrastructure to train models of this size efficiently before.

Hardware advances

The modern Deep Learning / AI craze started in 2012, when a neural network called AlexNet won the imagenet challenge. The architecture used a convolutional neural network, a method that had been invented 25 years prior and had been deemed too impractical to use. What changed?

The short answer? GPUs happened. Modern graphics applications had made specialized hardware for linear algebra cheap and commercially available. Chips had gotten almost a thousand times faster during that period, following Moore's law. When combined with myriad other computing advances in areas such as memory and network interfaces, it might not be a stretch to say that the GPU which AlexNet ran on was a million times better for Convnets than what had been available when Convnets were invented.

As the craze took off, NVIDIA stared optimizing their GPUs more and more for deep learning workloads across the stack. GPUs were given more memory to hold larger models and more intermediate computations, faster interconnects to leverage multiple gpus at once, and more optimized primitives through CUDA and cuDNN. This enabled much larger models, but by itself would not have allowed for the absolutely giant models we see now.

Software advances

In the old days of deep learning, linear algebra operations had to be done by hand... well not really, but programming was a pain and the resulting programs used resources poorly. Switching to a GPU was a nightmare, and if trying to use multiple GPUs would make a pope question their faith in god. Then along came Caffe, then tensorflow, then pytorch, and suddenly training was so easy that any moron with an internet connection (me!) can use deep learning without understanding any of the math, hardware, or programming that needs to happen.

These days, training an ML model doesn't even require coding knowledge. If you do code, your code probably works on CPUs or GPUs, locally or on AWS/Azure/Google Cloud, and with one GPU or 32 across four different machines.

Furthermore, modern ML platforms will do under the hood optimization to accelerate model execution. Easy to write ML models now are easy to share, readable, well optimized, and can be scaled easily.

Research innovations

There are two sets of important advances that enabled large scale research. The first are a legion of improvements to methods that allowed for less computational to achieve more. The second are methods that allow for more computation to be thrown at the same problem for better and faster results. While there are also many advances in this field, three stick out: transformers, pipeline parallelism, and self supervised learning

Transformers are models that run really well on GPUs by leveraging very efficient matrix multiplication primitives. They were originally designed for text data, but it turns out that for models whose size is measured in gigabytes, transformers are just better than their competition for the same amount of computation.

If we think back to our weightlifting analogy, transformers are like your leg muscles. For sufficiently large loads, they can't be beat!

(As an ML Systems researcher, every new application where just throwing a big transformer at a problem beats years of custom machine learning approaches and panels of experts brings me a strange joy.)

Pipeline parallelism is a bit complicated to explain to a nontechnical audience, but the short version is that training a machine learning model requires much more memory on the GPU than the size of the model. For small models, splitting the data between GPUs is the best approach. For large models, splitting the model across GPUs is better. Pipeline parallelism is a much better way of splitting the model than prior approaches, especially for models which are larger than a gigabyte.

Pipeline parallelism is like having good form. For smaller lifts its not a big deal, but it is critical for larger lifts.

Self supervised learning is like making flashcards to study for a test. Ideally, your teacher would make you a comprehensive set of practice questions, but that takes a lot of effort for the teacher. A self directed student could take data that doesn't have "questions" (labels) and make up your own questions to learn the material. For example, a model trying to learn English could take a sentence, hide a bunch of words, and try to guess them. This is much cheaper than having a human "make questions".

Self supervised learning is like cooking your own food instead of hiring a personal chef for your nutritional needs. It might be worse, but it is so much cheaper, and for the same price you can make a lot more food!

How have those underlying factors changed in the past three years?

TL;DR not much. We haven't gotten stronger in the past four years, just did a bunch of different exercises which used the same muscles.

All of the advances I mentioned in the last section were from 2018 or earlier.

(For the purists, Self supervised learning went mainstream for vision in 2020 by finally outperforming supervised learning).

Chips are not getting twice as fast every two years like they used to (Moore's law is dying). The cost of a single training run for the largest ML models is on the order of ten million dollars. Adding more GPUs and more computation is pushing against the amount that companies are willing to burn on services that don't generate money for the company. Unlike the prior four years, we cannot scale up the size of models by a thousand times again. No one is willing to spend billions of dollars on training runs yet.

From a hardware perspective, we should expect the pace of innovation to slow in the coming years.

Software advances are mixed. Using ML models is becoming easier by the day. With libraries like huggingface, a single line of code can run a state of the art model for your particular use case. There is a lot of room for software innovations to make it easier to use for non technical audiences, but right now very little research is bottlenecked by software.

Research advances are the X factor. Lots of people are working on these problems, and its possible there is a magic trick for intelligence at existing compute budgets. However, that is and always was true. However, the most important research advances of the last few years primarily enabled us to use more GPUs for a given problem. Now that we are starting to run up against the limits of data acquisition and monetary cost, less low hanging fruit is available.

(Side note: Even facebook has trouble training current state of the art models. Here are some chronicles of them trying to train a GPT-3 size model).

Conclusion.

I don't think we should expect performance gains in AI to accelerate over the next few years. As a researcher in the field, I expect the next few years will involve a lot of advances in the "long tail" of use cases and have less growth in the most studied areas. This is because we have achieved the easy pickings gains from hardware and software over the past decade.

This is my first time posting to lesswrong, and I decided to post a lightly edited first draft because if I start doing heavy edits I don't stop. Every time I see a very fast AGI prediction or someone claiming Moore's law will last a few more decades I start to write something, but this time I actually finished it before deciding to rewrite. As a result, it isn't an airtight argument, but more my general feelings as someone who has been at two of the top research institutions in the world.



Discuss

Reading the ethicists: A review of articles on AI in the journal Science and Engineering Ethics

Новости LessWrong.com - 18 мая, 2022 - 23:52
Published on May 18, 2022 8:52 PM GMT

Epistemic status: Stream of consciousness reactions to papers read in chronological order. Caveat lector.

I have a dirty family secret. My uncle is a professional ethicist.

In a not-too roundabout way, this is why I ended up looking at the October 2020 issue of the journal Science and Engineering Ethics, their special issue on the ethics of AI. I am now going to read that issue, plus every article this journal has published about AI since then [I wussed out and am just going to skim the latter for ones of special interest] and give you the deets.

October 2020Hildt et al., Editorial: Shaping Ethical Futures in Brain-Based and Artificial Intelligence Research

This is the introduction to the issue. They give each paper a sentence or two of summary and try to tie them all together. The authors helpfully give a list of topics they think are important:

• Data Concerns: Data management, data security, protection of personal data, surveillance, privacy, and informed consent.

• Algorithmic Bias and Discrimination: How to avoid bias and bias related problems? This points to questions of justice, equitable access to resources, and digital divide.

• Autonomy: When and how is AI autonomous, what are the characteristics of autonomous AI? How to develop rules for autonomous vehicles?

• Responsibility: Who is in control? Who is responsible or accountable for decisions made by AI?

• Questions relating to AI capabilities: Can AI ever be conscious or sentient? What would conscious or sentient AI imply?

• Values and morality: How to build in values and moral decision-making to AI? Are moral machines possible? Should robots be granted moral status or rights?

Based on this list, I anticipate that I'm about to run into four-sixths ethics papers about present-day topics that I will skim to point out particularly insightful or anti-insightful ones, one-sixth philosophers of mind that I will make fun of a little, and one-sixth papers on "How to build values into general AI" that I'm really curious as to the quality of.

Onward!

Nallur, Landscape of Machine Implemented Ethics

Primarily this paper is a review of a bunch of papers that have implemented or proposed ethics modules in AI systems (present-day things like expert systems to give medical advice, or lethal autonomous weapons [which he has surprisingly few qualms about]). These were mostly different varieties of rule-following or constraint-satisfaction, with a few handwritten utility functions thrown in. And then one of these is Stuart Armstrong (2015) for some reason - potentially that reason is that the author wanted to at least mention "value-loading," and nobody else was talking about it (I checked - there's a big table of properties of different proposals).

It also proposes evaluating different proposals by having a benchmark of trolley-problem-esque ethical dilemmas. The main reason this idea won't work is that making modern-day systems behave ethically involves a bunch of bespoke solutions only suitable to the domain of operation of that system, not allowing for cross-comparison in any useful way.

If were to salvage this idea, we might wish to have a big list of ethical questions the AI system should get the right answer to, and then when building a sufficiently important AI (still talking about present-day applications), the designers should go through this list and find all the questions that can be translated into their system's ontology and check that their decision-making procedure gets acceptable answers. E.g. "Is it better to kill one person or two people?" can become self-driving car scenarios where it's going to hit either one or two people, and it should get the right answer, but the self-driving car people don't have to benchmark their system on medical ethics questions.

Bauer, Expanding Nallur's Landscape of Machine Implemented Ethics

This paper makes the point that "ethical behavior" for present-day AI might also mean taking a stand about how we want society to be arranged on a larger scale (e.g. what's an "ethical high-frequency trading algorithm"?). Then it descends into self-advertisement about hedonistic utilitarianism and virtue ethics, which we should clearly not build AIs to use.

Farisco et al., Towards Establishing Criteria for the Ethical Analysis of Artificial Intelligence

This one makes a lot of the right noises at first. They spend a couple of pages talking about defining intelligence for some unclear reason, but at least they cite Legg and Hutter, you know? But then they quickly take a left turn off the deep end and start talking about how biological intelligence is morally different than AI because AI can't do abductive reasoning. There are some good points about emotional connection mixed in with the bad points about emotional reasoning being magic, I guess.

Butkus, The Human Side of Artificial Intelligence

This is a response to the previous article, especially the parts about emotional reasoning being magic. The key point it makes (a good one) is that humans aren't all that great at reasoning - we make lots of mistakes, including moral mistakes. "If we intentionally avoid some of the known pitfalls in our cognitive architecture, we cannot help but create moral agents that are dissimilar from us." How to do this in a trustworthy way is obviously hard, and they mumble something about contextuality of decision making.

Rainey and Erden, Correcting the Brain? The Convergence of Neuroscience, Neurotechnology, Psychiatry, and Artificial Intelligence

We might use AI to control brain stimulation to try to treat people with psychiatric problems. This would be like the AI controlling people, which is scary. Those darn neuroscientists are too reductive about the brain. Boo reductionism, yay holism. Humans look for and can relay bite-sized reasons for their actions, while AIs can't, which is why human reasoning is more trustworthy. </they say>

Jotterand and Bosco, Keeping the “Human in the Loop” in the Age of Artificial Intelligence

Props to this paper for having a good abstract. It's about risks from "dehumanizing" medicine. Their best point is that there's an inequality in the doctor-patient relationship, but that part of good medicine is establishing trust with the patient and genuinely working together with them to answer moral/medical questions. Of course they then say "AI will never ever be able to do this," but we can charitably interpret them as saying that AI can do it neither now nor soon, and that there are dangerous incentives to hastily replace doctors with AI in a way that damages patients' trust and agency.

Dubljević, Toward Implementing the ADC Model of Moral Judgment in Autonomous Vehicles

ADC stands for "Agent, Deed, Consequence." This would evaluate actions using a nearly-equal mixture of three parts. Normally the "Agent" part means that an action is more moral if the agent was a good person with good intentions, but in this paper the author also gives it the job of making actions more moral if you're helping good people or harming bad people. (Does this make any sense? Especially to program into a self-driving car? No, this seems like mixing up descriptive and normative.). "Deed" means checking if the action obeys the law or other pre-specified rules of the road. "Consequence" means not crashing or causing crashes, and getting to the destination.

The author gives a cherry-picked terrorism example where self-driving cars are supposed to notice that a terrorist is driving a truck into a crowd, judge that they're a bad person, and then will not get out of the truck's way because "Action" is negative (getting out of the way would help a bad person), "Deed" is negative (you'd have to leave your lane to avoid the truck, which is against the rules), and only "Consequence" is positive (you'd avoid crashing).

This isn't totally insane. We could imagine e.g. giving self-driving cars rules for identifying when another car is about to crash into people and how to try to intercept it. But I think this notion of needing a triumvirate where the three votes are Law, Not Crashing, and Harming Bad People is just the wrong design desiderata.

Totschnig, Fully Autonomous AI

Our first paper about ethics for par-human and superhuman AI! They care about something they call "full autonomy," where you sometimes rewrite your own goals, which humans sort of do (the example is someone who devotes themself to one cause but then later changes their mind). They then give a good summary of why Bostrom And The Gang think that an AI won't want to change its goals (which they call the Finality Argument that a self-improving AI's goals will remain fixed).

My first response was that humans only sort of have goals, and therefore only sort of rewrite them. This is anticipated in the next section, and the author basically says this is a good point, but they still think "full autonomy" is important and in some sense desirable. *shrug*

Their knock-down counterargument to the Finality Argument is that improving your model of the world requires you to translate your old goals into the new ontology, a task which is not specified by the goals themselves but by some extra standards (that may themselves be subject to change.) This is rewriting your goals, and so self-improving AIs have full autonomy as they define it.

All valid so far. But then they conclude "The good news is that the fear of a paper clip AI and similar monsters is unfounded. The bad news is that the hope of a human-equivalent or superhuman AI under our control, of a genie in a bottle, is unfounded as well." They think AIs are going to search for moral realism and end up having weird and surprising goals. But this makes an unsupported leap from the "full autonomy" they care about to "will radically rather than only subtly change itself."

One more thing I'm interested in here is to look through the references to see if there are any famous (well, at least well-cited) ethicists writing about things like this who I haven't heard of. A lot of the references were to the usual suspects (Bostrom, Yudkowsky, Russell, Yampolskiy, etc.). Papers for me to look into:

Lawless, W. F., Mittu, R., Sofge, D., & Russell, S. (Eds.). (2017). Autonomy and artificial intelligence: A threat or savior? (A volume of collected papers)

Redfeld, S. A., & Seto, M. L. (2017). Verification challenges for autonomous systems.

Tessier, C. (2017). Robots autonomy: Some technical issues.

Petersen, S. (2017). Superintelligence as superethical. In P. Lin, R. Jenkins, & K. Abney (Eds.), Robot ethics 2.0: From autonomous cars to artificial intelligence (pp. 322–337).

Podschwadek, F. (2017). Do androids dream of normative endorsement? On the fallibility of artificial moral agents. Artificial Intelligence and Law, 25(3), 325–339

I'll get back to these later.

Dennis, Computational Goals, Values and Decision‑Making

This is commentary on the previous paper. It mostly says "hold up, you can't actually implement an agent with a utility function, you need to use some kind of bounded rationality" without drawing strong conclusions.

Matthias, Dignity and Dissent in Humans and Non‑humans

This paper is another one that intensely cares about whether AIs (and animals) have autonomy and/or the ability to give themselves goals. But this time it explains some context: Kant cared a lot about human dignity and "derives it from the moral autonomy of the individual," and so people who've drunk the Kant kool-aid (probably could phrase that better, oh well) are asking if AIs have moral autonomy so they can know whether they're dignified.

However, this paper quickly decides that all that matters is autonomy similar in kind and quantity to that of a human, which they say isn't all that high a bar. However, they also say it isn't "real choice" if the AI is following steps for moral reasoning laid down by its programmers, which seems to mix up rule-goverenedness with human understanding of those rules (appeal to the mystique of mystery).

There's also something about individuality being important to them, which sort of makes sense but also sort of sounds like the author preferentially thinks about human-sized and human-shaped entities.

Zhu et al., Blame‑Laden Moral Rebukes and the Morally Competent Robot: A Confucian Ethical Perspective

Many of these authors have been from not-so-famous universities (I'm still used to the hard sciences where funding differential and race dynamics means that the same handful of institutions reliably dominate), but this one is surprising enough to mention: the authors are from the Colorado School of Mines, which I wasn't even aware had a school of humanities (they're a decent but very focused engineering school).

This paper is about an interesting question: should near-future language models (or AI systems that use a language model) output "blame-laden moral rebukes" of users who misbehave? If you start swearing at the AI receptionist, should it tell you off?

The authors first give some low-sample-size evidence that humans will be influenced by AI enforcement of moral norms (with different forms of enforcement, ranging from polite mentions to blunt demands, working better in different contexts). Then they spend a section explaining some facets of Confucianism that relate to when you should rebuke others, which point to some of the theory of mind an effectively-rebuking AI should have. They also advocate for some Confucian norms about filtering communication through social role.

Gunkel, Shifting Perspectives

This is commentary on the previous article. It's not very good, but does have some nice references if you want to look into the subfield called Human-Machine Communication.

Aicardi et al., Ethical and Social Aspects of Neurorobotics

This article comes out of the Human Brain Project, an ambitious (and non-ambition-meeting) effort to scan the human brain. After getting what felt like boilerplate social concerns out of the way, they share some of the things that they had to think about the ethics of at HBP. The key sections were on the dual use potential of robotics research, the issues with academic-industry partnerships, and managing and securing your data.

Taraban, Limits of Neural Computation in Humans and Machines

This was supposed to be commentary on the previous article, except the author didn't want to talk about research ethics, they wanted to talk about whether it makes sense to build robots using AI derived from scans of human brains.

First, they say it's silly in the first place. Humans are flawed thinkers, and anyhow we don't know how to take a brain scan and turn it into an AI with customized semantic content, and anyhow the probabilistic inference people are probably going to build better robots first.

Second, if the neurorobotics people succeed we'd probably grant the robots rights, and that would be silly so let's not do it.

Soltanzadeh et al., Customizable Ethics Settings for Building Resilience and Narrowing the Responsibility Gap

This is an article about autonomous vehicles, arguing that they should come with user-modifiable "ethics settings." These could be things like setting the tradeoff between speed and greenhouse gas emissions, adjusting car follow distance and pedestrian avoidance distance within some range, etc.

Basically they call user autonomy a "core social value," and also worry that if the human in the vehicle isn't responsible for what the car does, that's inherently bad. The weirdness of these arguments actually makes me re-examine my intuitive agreement with the idea of ethics settings.

Ryan, In AI We Trust: Ethics, Artificial Intelligence, and Reliability

The EU's HLEG AI Ethics guidelines state that humans should be able to trust AI. The author splits hairs, gives some definitions of trust that require emotional states or motivations on the part of the person being trusted, and then concludes that we don't actually want to be able to trust AI, we just want to be able to rely on it. *shrug*

Smids, Danaher’s Ethical Behaviourism: An Adequate Guide to Assessing the Moral Status of a Robot?

This is a response to an apparently-popular paper, Danaher 2019, that argues for "ethical behaviorism" for robots - if they act like an animal, then we should ascribe animal-like moral status to them. The author (Smids) disagrees.

This paper could really use more Bayesian reasoning. They try to muddle through based on what is qualitatively "justified" in abductive reasoning, but it gets real murky.

The point the author makes is: ethical behaviorism is sneaking in a theory about what makes things have moral status, and if we just use that theory we don't need to keep the behaviorism part.

For example, breathing is a behavior that many animals of high moral status do, but we don't ask a robot to breathe before it can have moral status. But how do we know to discard breathing, if we're just being behaviorists? We're discriminating between different kinds of behavior by checking how much information they give us about the morally significant properties (e.g. feeling pain, having dreams) that we actually care about.

And then once you allow that we're using some theory about mental faculties and inferring them from behavior, it makes sense to allow us to infer them from other things too - surely the design of a robot gives us more than literally zero additional information about its mental faculties.

I think this paper was reasonable and well argued, but I think it also illuminates some issues in ethicists' approach to AI. First, Danaher's paper feels like it might have been popular because of the "style points for cleverly defending something silly" dynamic that's occasionally a problem, plus some wishful thinking for simple solutions. Second, both the original paper and the response focus on animal-like or human-like AI controlling individual robot bodies, a very anthropomorphic (zoomorphic?) picture that seems to show up repeatedly whenever people first try to think about the "moral worth of AI."

Cawthorne and van Wysnberghe, An Ethical Framework for the Design, Development, Implementation, and Assessment of Drones Used in Public Healthcare

This paper foresees an expansion of drone usage by healthcare organizations and tries to go through ethical principles (beneficence, non-maleficence, autonomy, justice, explicability) to try to draw recommendations. Among many recommendations of varying quality, some of the interesting ones were:

  • Deliberately limiting the drone's capabilities to only what's needed for its specific job (avoiding excess capability)
  • Taking steps to make sure that it's really hard to use them for surveillance
  • Flying them along designated corridors rather than the shortest route if that aids predictability and helps avoid crashes
  • Making sure they're quiet
  • Technological transparency
Later Papers I Found InterestingHubbard and Greenblum, Surrogates and Artificial Intelligence: Why AI Trumps Family (December 2020)

When someone is incapacitated, right now we let their family make medical decisions for them. Instead, we should train a model of medical decision-making on the choices of non-incapacitated patients, conditional on various demographic observables (age, sex, etc.) and use the predicted decision. If this model is more accurate at predicting the person's preferences than the family, then it's more ethical to use the model than to listen to the family.

A+ provocative paper.

Ryan et al., Research and Practice of AI Ethics: A Case Study Approach (March 2021)

Half the reason I include this is that they do devote a paragraph or two to "long-term issues" like superintelligent AI. But then they go back to looking for ethical issues in present-day case studies, and find only present-day type issues. Still, this was actually a pretty decent way of getting at peoples' concerns about present-day uses of AI and big data, which is the other half a reason to mention this paper.

Mamak, Rights for Robots: Artificial Intelligence, Animal and Environmental Law (2020) by Joshua Gellers (April 2021)

This is just a reminder that "Will robots have rights?" is absolute catnip to many ethicists. I will no longer be including more "But what about robot rights" papers, but you can be assured that they exist.

Lara, Why a Virtual Assistant for Moral Enhancement When We
Could have a Socrates? (June 2021)

I think I've actually skimmed this paper before. Or maybe the author's been shopping this idea around for a while.

As the title says, the author proposes a method (the method, they say) for augmenting humans morally, which is to help them reflect with non-invasive tools or partners that focus on improving their reasoning skills, not on leading them to one particular conclusion. Such a tool, let's call it "SocrAItes," might be possible quite soon, based on technical achievements like GPT3 or IBM's Project Debater.

I think it's an interesting idea that someone genuinely should try out, but I'm not sold that all other ideas are bad. Also, the author doesn't really think through how one would design or train such a system, so if you wanted to take a crack at it you'll need to start from square one.

de Sio, Mark Coeckelbergh, AI Ethics, Mit Press, 2021 (August 2021)

This book review wasn't that great, but it did make me go look up the original to see what Coeckelbergh had to say about superintelligent AI. Sadly, his chapter on superintelligence (the first chapter of the book) is spent rehashing Frankenstein and the myth of the golem, rather than talking about the ethics of superintelligent AI. Then the next chapter is spent resurrecting Hubert Dreyfus to argue against the possibility of general AI (plus a mildly interesting discussion of humanism, transhumanism, and anti-humanistic posthumanism.) But we get down to brass tacks in chapter 3, which is about whether robots should have rights.

Roberts et al., Achieving a ‘Good AI Society’: Comparing the Aims
and Progress of the EU and the US (November 2021)

An interesting summary paper if you want to read about international AI governance aimed at present-day issues. However, actually getting into the details of empirical questions like how the EU's AI regulations are actually being enforced seems like it's difficult and requires more data than is used in this paper - this paper mostly just covers the aims of the EU and US.

The gist of their picture is that the EU is trying to address a broad set of risks from current AI systems, and the US is trying to address a much more narrow set and is pressuring other countries to do the same because addressing more risks would cut into US company profits.

Schmid et al., Dual‑Use and Trustworthy? A Mixed Methods Analysis of AI
Diffusion Between Civilian and Defense R&D (January 2022)

This is an interesting attempt to characterize the dual-use of AI technology by looking at patent citations. Among 109 AI patent citations between different companies in Germany from 2008 to 2018, 93 stayed between civilian companies, 12 were a civilian company being cited by a defense company, 3 were a defense company cited by a civilian company, and 1 was between defense companies.

Which is interesting and all, but they don't actually do a good enough job of checking what this means for dual use (they say it's not happening). Like, how does this compare to citation patterns for technologies that more clearly are / are not dual use? Overall grade: cool idea, but I still have as many questions as I did at the start.

Conclusions

I did this so you don't have to. Out of the 45+ papers I looked through, I would say to read Nallur (2020) to get a survey of present-day machine ethics work, read Roberts et al. (2021) if you're interested in AI governance, and forward Hubbard and Greenblum (2020) to your doctor friends because it's a great troll.

There were fewer papers than I expected focused on the ethics of superhuman AI, though a decent number mentioned the issue (citing Bostrom, not really each other). However, I have found some good papers on the ethics of par-human or superhuman AI outside the journal Science and Engineering Ethics, which I'll cover in the sequel to this post. I'm not sure why this is - it could be that the fraction of ethics papers on superintelligence is constant and I merely found them effectively when I searched for them, or there was a bump in interest after the publication of Superintelligence: Paths, Dangers, Strategies that has now subsided, or this is due to a feature of the journal like its culture or an opinion of the editors.

What do I want to see in relation to ethicists? I don't think you can take someone who currently thinks about the average of the papers above, and immediately get them to have interesting thoughts about superhuman AI. But people can learn new things, and people interested in ethics are more or less the sort of people who'll get interested in the ethics of superhuman AI. So I would recommend more high-quality papers making the basic arguments in new ways or showing off incremental progress in a way accessible to the ethics or philosophy communities, but not recommend rapid incentives for new papers from people who haven't "put in the time."

One place that the current literature does seem relatively expert (and relatively influential) is in present-day governance of AI. I think that people working on AI governance with an eye towards risks from advanced AI should absolutely be trying to work in partnership with the broader AI regulatory community.



Discuss

Maxent and Abstractions: Current Best Arguments

Новости LessWrong.com - 18 мая, 2022 - 22:54
Published on May 18, 2022 7:54 PM GMT

This post is not-very-distilled and doesn’t contain much background; it’s intended for people who already have the context of at least these four posts. I’m putting it up mainly as a reference for people who might want to work directly on the math of natural abstractions, and as a technical reference post.

There’s various hints that, in most real-world cases, the distribution of low-level state given high-level natural abstractions should take the form of a maximum entropy distribution, in which:

  • The “features” are sums over local terms, and
  • The high-level variables are (isomorphic to) the Lagrange multipliers

More formally: we have a low-level causal model (aka Bayes net) P[XL]=∏iP[XLi|XLpa(i)].mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} . Given the high-level variables XH, the distribution of low-level variable values should look like

P[XL|XH]=1ZP[XL]eλT(XH)∑ifi(XLi,XLpa(i))

… i.e. the maximum-entropy distribution subject to constraints of the form E[∑ifi(XLi,XLpa(i))|XH]=μ(XH). (Note: λ, fi, and μ are all vector-valued.)

This is the sort of form we see in statistical mechanics. It’s also the form which the generalized Koopman-Pitman-Darmois (gKPD) theorem seems to hint at.

I don’t yet have a fully-satisfying general argument that this is the main form which abstractions should take, but I have two partial arguments. This post will go over both of them.

Maxent Telephone Theorem ArgumentTwo different nested layers of Markov blankets on the same underlying causal DAG

Quick recap of the Telephone Theorem: information about some variable X passes through a nested sequence of Markov blankets M1,M2,…. Information about X can only be lost as it propagates. In the limit, all information is either perfectly conserved or completely lost. Mathematically, in the limit P[X|Mn]=P[X|Fn(Mn)] for some F such that Fn(Mn)=Fn+1(Mn+1) with probability approaching 1 as n→∞; F is the perfectly-conserved-in-the-limit information carrier.

In this setup, we can also argue that the limiting distribution limn→∞P[X|Mn] should have a maxent form. (Note: this is a hand-wavy argument, not a proper proof.)

Think about how the distribution (x↦P[X=x|Mn]) transforms as we increment n by 1. We have

P[X|Mn+1]=∑MnP[X|Mn]P[Mn|Mn+1]

First key property of this transformation: it’s a convex combination for each Mn+1 value, i.e. it’s mixing. Mixing, in general, cannot decrease the entropy of a distribution, only increase it or leave it the same. So, the entropy of P[X|Mn] will not decrease with n.

When will the entropy stay the same? Well, our transformation may perfectly conserve some quantities. Since the transformation is linear, those quantities should have the form ∑Xf(X)P[X|Mn] for some f, i.e. they’re expected values. They’re conserved when E[f(X)|Mn]=E[f(X)|Mn+1] with probability 1.

Intuitively, we’d expect the entropy of everything except the conserved quantities to strictly increase. So, we’d expect the distribution P[X|Mn] to approach maximum entropy subject to constraints of the form E[f(X)|Mn]=μ(Mn), where E[f(X)|Mn]=E[f(X)|Mn+1] with probability 1 (at least in the limit of large n). Thus, we have the maxent form

P[X|Mn]=1ZP[X]eλT(Mn)f(X)

(Note on the P[X] in there: I’m actually maximizing relative entropy, relative to the prior on X, which is almost always what one should actually do when maximizing entropy. That results in a P[X] term. We should find that E[lnP[X]|Mn] is a conserved quantity anyway, so it shouldn’t actually matter whether we include the P[X] multiplier or not; we’ll get the same answer either way.)

Shortcomings of This Argument

Obviously it’s a bit handwavy. Other than that, the main issue is that the Telephone Theorem doesn’t really leverage the spatial distribution of information; information only propagates along a single dimension. As a result, there’s not really a way to talk about the conserved f’s being a sum over local terms, i.e. f(X)=∑ifi(Xi,Xpa(i)).

Despite the handwaviness, it’s an easy result to verify computationally for small systems, and I have checked that it works.

Resampling + gKPD Argument

Another approach is to start from the redundancy + resampling formulation of abstractions. In this approach, we run an MCMC process on our causal model. Any information which is highly redundant in the system - i.e. the natural abstractions - is near-perfectly conserved under resampling a single variable at a time; other information is all wiped out. Call the initial (low-level) state of the MCMC process X0, and the final state X. Then we have

P[X|X0]=P[X|F(X0)]=P[X|F(X)]P[F(X)|F(X0)]=1ZP[X]I[F(X)=F(X0)]

… where F is conserved by the resampling process with probability 1.

It turns out that P[X|X0] factors over the same DAG as the underlying causal model:

P[X|X0]=∏iP[Xi|Xpa(i),X0]

If the conserved quantities F(X) are much lower-dimensional than X itself, then we can apply the gKPD theorem: we have a factorization of P[X|X0], we have a low-dimensional summary statistic F(X) which summarizes all the info in X relevant to X0, so the gKPD theorem says that the distribution must have the form

P[X|X0]=1ZeλT(X0)∑i∉Efi(Xi,Xpa(i))∏i∉EP[Xi|Xpa(i),X0=(X0)∗]∏i∈EP[Xi|Xpa(i),X0=X0]

… where E is a relatively-small set of “exceptional” indices, and (X0)∗ is some fixed reference value of X0. This is slightly different from our intended form - there’s the exception terms, and we have ∏i∉EP[Xi|Xpa(i),X0=(X0)∗] rather than just ∏i∉EP[Xi|Xpa(i)]. The latter problem is easily fixed by absorbing ∏i∉EP[Xi|Xpa(i),X0=(X0)∗]P[Xi|Xpa(i)] into f (at the cost of possibly increasing the summary dimension by 1), so that’s not really an issue, but the exception terms are annoying. Absorbing and assuming (for convenience) no exception terms, we get the desired form:

P[X|X0]=1ZeλT(X0)∑ifi(Xi,Xpa(i))P[X]

Note that this is maxentropic subject to constraints of the form E[∑ifi(Xi,Xpa(i))|X0]=μ(X0). Since the summary statistic F(X)=∑ifi(Xi,Xpa(i)) is conserved by the resampling process, we must have μ(X0)=∑ifi(X0i,X0pa(i)), so the conservation equation is

E[∑ifi(Xi,Xpa(i))|X0]=∑ifi(X0i,X0pa(i))

Shortcomings of This Argument

Obviously there’s the exception terms. Other than that, the main issue with this argument is an issue with the resampling approach more generally: once we allow approximation, it’s not clear that the natural abstractions from the resampling formulation are the same natural abstractions which make the Telephone Theorem work. Both are independently useful: information dropping to zero at a distance is an easy property to leverage for planning/inference, and knowing the quantities conserved by MCMC makes MCMC-based planning and inference much more scalable. And in the limit of perfect conservation and infinite “distance”, the two match. But it’s not clear whether they match under realistic approximations, and I don’t yet have efficient methods to compute the natural abstractions both ways in large systems in order to check.

That said, resampling + gKPD does give us basically the result we want, at least for redundancy/resampling-based natural abstractions.



Discuss

Framing Practicum: Dynamic Programming

Новости LessWrong.com - 18 мая, 2022 - 21:16
Published on May 18, 2022 6:16 PM GMT

This is a framing practicum post. We'll talk about what dynamic programming (DP) is, how to recognize DP in the wild, and what questions to ask when you find it. Then, we'll have a challenge to apply the idea.

Today's challenge: come up with 3 examples of DP which do not resemble any you've seen before. They don't need to be good, they don't need to be useful, they just need to be novel (to you).

Expected time: ~15-30 minutes at most, including the Bonus Exercise.

What is DP?

Suppose I am about to drive from Miami to Boston and I need to get to Boston as fast as possible. As a first step, I check the highway map and create a list of possible routes for this trip (let’s assume “good” old times with no Google maps). For instance, looking at the imaginary routes in the figure below, the route at the top says I should take the “Miami → A → C → E → G → Boston” route and the total trip distance would be 200+170+220+400+430=1420 miles. Which specific route should I take to minimize the total distance, thus total travel time? I can, of course, calculate total travel distance for each possible route and pick the one with least-distance. But it could easily get very time consuming if there exist hundreds of thousands of possible routes to evaluate.

One alternative approach to identify a route with minimum distance is to use a backward method. Suppose I drive backward through the route map from right to left. First, I will start at the destination, Boston. If I am in city G or H, I have no further decision to make since there is only one route that leads me to Boston from either city. The number in green summarizes total distance with one last trip, or one stage, to go.

My first decision (from right to left) occurs with two trips, or stages to go. If, for example, I am in city F, I can either drive 150 miles to city G and another 430 miles to Boston - total 580 miles, or drive 200 miles to city H and another 400 miles to Boston - total 600 miles. Therefore, the shortest possible distance, or optimal route (in green), from city F is 580 miles (F → G → Boston). The alternative route from F (F → H → Boston) is suboptimal with 600 miles to go (in red). 

Let me back up one more city, or stage, and compute the least-distance from city C and D to Boston. Figure below summarizes these calculations.

Once I have computed the optimal route from city C and D onward to Boston, I can again move back one city and determine the optimal route from city A and B onward.

I continue this process and will end up with an optimal route with least-distance of 1080 miles to the problem (highlighted in bold arrows):

This is DP: An agent faces a multistage optimization problem (travelling from Miami to Boston by travelling through multiple cities). At each stage (e.g., I have one more trip to go to Boston), the agent might be in a different state (e.g., I am currently in city A or city B). According to the current state (e.g., I am in city A), the agent takes a specific action (e.g., I will drive to city C) and as a result of that action, the system transitions to a different state in the next stage (e.g., now I am in city C). We solve the multistage optimization problem by working backwards, at each step computing the best reward we can get from that stage onward by considering each of the possible “next/child” stages.

What To Look For?

DP should come to mind whenever an agent faces a problem with multi-stages nature and the agent takes a series of actions. Another defining feature of DP is that the original multi-stage complex problem can be dismantled into a sequence of simpler and smaller problems. The action the agent takes in a particular stage depends on the current state and the reward the agent would receive by taking that specific action. In addition, the action the agent takes impacts the state of the system, causing the system to transition to a new state.

Useful Questions to Ask?

In the shortest driving time example, the ultimate goal is to minimize total driving time such that I can arrive at Boston as fast as possible. At any given time in my trip, I might be in a different state - the city I am in at that time. For instance, on the second day of the trip, I might be in city C or city D. Given my state,  I have a simpler problem to solve: What is the shortest travel time from city C or city D to Boston?

The system may start in different states. The agent takes a series of actions to optimize an objective across multiple stages. Each stage also has multiple states. The specific action an agent can take is a function of the state the agent currently in.

In general, whenever we see problems where DP is applicable, we should ask:

  • What is the objective?
  • What are the stages and states of the system? 
  • What are the actions the agent can take at any given state?
  • How does a specific action change the state of the system?
  • What is the value function? How is an agent rewarded for taking a particular action at the current stage?
The Challenge

Come up with 3 examples of incentives which do not resemble any you’ve seen before. They don’t need to be good, they don’t need to be useful, they just need to be novel (to you).

Any answer must include at least 3 to count, and they must be novel to you. That’s the challenge. We’re here to challenge ourselves, not just review examples we already know.

However, they don’t have to be very good answers or even correct answers. Posting wrong things on the internet is scary, but a very fast way to learn, and I will enforce a high bar for kindness in response-comments. I will personally default to upvoting every complete answer, even if parts of it are wrong, and I encourage others to do the same.

Post your answers inside of spoiler tags. (How do I do that?)

Celebrate others’ answers. This is really important, especially for tougher questions. Sharing exercises in public is a scary experience. I don’t want people to leave this having back-chained the experience “If I go outside my comfort zone, people will look down on me”. So be generous with those upvotes. I certainly will be.

If you comment on someone else’s answers, focus on making exciting, novel ideas work — instead of tearing apart worse ideas. Yes, And is encouraged.

I will remove comments which I deem insufficiently kind, even if I believe they are valuable comments. I want people to feel encouraged to try and fail here, and that means enforcing nicer norms than usual. 

If you get stuck, look for:

  • Problems with multistage nature.
  • Problems that can be dismantled into a sequence of simpler problems.

Bonus Exercise: for each of your three examples from the challenge, explain:

  • What are the simpler and smaller problems in the DP example?
  • What are the states and how taking a specific action alters the state?
  • what is the reward for taking a specific action on a given state?

This bonus exercise is great blog-post fodder!

Motivation

Using a framing tool is sort of like using a trigger-action pattern: the hard part is to notice a pattern, a place where a particular tool can apply (the “trigger”). Once we notice the pattern, it suggests certain questions or approximations (the “action”). This challenge is meant to train the trigger-step: we look for novel examples to ingrain the abstract trigger pattern (separate from examples/contexts we already know).

The Bonus Exercise is meant to train the action-step: apply whatever questions/approximations the frame suggests, in order to build the reflex of applying them when we notice incentives.

Hopefully, this will make it easier to notice when an incentive frame can be applied to a new problem you don’t understand in the wild, and to actually use it.



Discuss

How to get into AI safety research

Новости LessWrong.com - 18 мая, 2022 - 21:05
Published on May 18, 2022 6:05 PM GMT

Recently, I had a conversation with someone from a math background, asking how they could get into AI safety research. Based on my own path from mathematics to AI alignment, I recommended the following sources. It may prove useful to others contemplating a similar change in career:

  • Superintelligence by Nick Bostrom. It condenses all the main arguments for the power and the risk of AI, and gives a framework in which to think of the challenges and possibilities.
  • Sutton and Barto's Book: Reinforcement Learning: An Introduction. This gives the very basics of what ML researchers actually do all day, and is important for understanding more advanced concepts. It gives (most of) the vocabulary to understand what ML and AI papers are talking about.
  • Gödel without too many tears. This is how I managed to really grok logic and the completeness/incompleteness theorems. Important for understanding many of MIRI's and LessWrong's approaches to AI and decision theory.
  • Safely Interruptible agents. It feels bad to recommend one of my own papers, but I think this is an excellent example of bouncing between ML concepts and alignment concepts to make some traditional systems interruptible (so that we can shut them down without them resisting the shutdown).
  • Alignment for Advanced Machine Learning Systems. Helps give an overall perspective on different alignment methods, and some understanding of MIRI's view on the subject (for a deeper understanding, I recommend diving into MIRI's or Eliezer's publications/writings).

You mileage may vary, but these are the sources that I would recommend. And I encourage you to post any sources you'd recommend, in the comments.



Discuss

A bridge to Dath Ilan? Improved governance on the critical path to AI alignment.

Новости LessWrong.com - 18 мая, 2022 - 18:51
Published on May 18, 2022 3:51 PM GMT

Summary: This post showcases my finalist entry in the Future of Life Institute's AI worldbuilding contest.  It imagines:

  1. How we might make big improvements to decisionmaking via mechanisms like futarchy and liquid democracy, enhanced by Elicit-like research/analysis tools.
  2. How changes could spread to many countries via competition to achieve faster growth than rivals, and via snowball effects of reform.
  3. How the resulting, more "adequate" civilization could recognize the threat posed by alignment and coordinate to solve the problem.

(Cross-posted to the EA Forum)

Part of a mural illustrating our scenario, created by Diana Gurvich!Motivation for our scenario:

Human civilization's current ability to coordinate on goals, make wise decisions quickly, and capably execute big projects, seems inadequate to handle the challenge of safely developing aligned AI.  Evidence for this statement can be found practically all around you, but the global reaction to covid-19 is especially clarifying.  Quoting Gwern:

The coronavirus was x-risk on easy mode: a risk (global influenza pandemic) warned of for many decades in advance, in highly specific detail, by respected & high-status people like Bill Gates, which was easy to understand with well-known historical precedents, fitting into standard human conceptions of risk, which could be planned & prepared for effectively at small expense, and whose absolute progress human by human could be recorded in real-time happening rather slowly over almost half a year while highly effective yet cheap countermeasures like travel bans & contact-tracing & hand-made masks could—and in some places did!—halt it. Yet, most of the world failed badly this test; and many entities like the CDC or FDA in the USA perversely exacerbated it, interpreted it through an identity politics lenses in willful denial of reality, obstructed responses to preserve their fief or eek out trivial economic benefits, prioritized maintaining the status quo & respectability, lied to the public “don’t worry, it can’t happen! go back to sleep” when there was still time to do something, and so on. If the worst-case AI x-risk happened, it would be hard for every reason that corona was easy. When we speak of “fast takeoffs”, I increasingly think we should clarify that apparently, a “fast takeoff” in terms of human coordination means any takeoff faster than ‘several decades’ will get inside our decision loops. Don’t count on our institutions to save anyone: they can’t even save themselves.

Around here on LessWrong, proposed AI x-risk-mitigation strategies generally attempt to route around this problem by aiming to first invent an aligned superintelligent AI, then use the superintelligent AI to execute a "pivotal action" that prevents rival unaligned AIs from emerging and generally brings humanity to a place of existential security.

This is a decent Plan A -- it requires solving alignment, but we have to solve that eventually in almost every successful scenario (including mine).  It doesn't require much else, making it a nice and simple plan.  One problem might be that executing a massive "pivotal action" might work less well if AI capabilities develop more smoothly and capabilities are distributed evenly among many actors, a la "slow takeoff" scenarios.

But some have argued have argued that we might be neglecting "Plan B" strategies built around global coordination.  The post "What An Actually Pessimistic Containment Strategy Looks Like" considers Israel's successful campaign to stop Iran from developing nuclear weapons, and argues that activist efforts to slow down AGI research at top tech companies might be similarly fruitful.  Usually (including in my worldbuilding scenario), it's imagined that the purpose of such coordination is to buy a little more time for technical alignment safety work to happen.  But for a more extreme vision of permanently suppressing AI technology, we can turn to the fictional world of Dath Ilan, or to Nick Bostrom's "easy nukes" thought experiment exploring how humanity could survive if nuclear weapons were absurdly easy to make.

The idea that we should push for improved governance in order to influence AI has its problems.  It takes a long time, making it might be very helpful in 2070 but not by 2030.  (In this respect it is similar to other longer-term interventions like gene-editing to create more scientific geniuses or general EA community-building investments.)  And of course you still have to solve the technical challenge of AI alignment in the end.  But improving governance also has a lot to recommend it, and it's something that can ideally be done in parallel with technical alignment research -- complementing rather than substituting, worked on by different people who have different strengths and interests.

Finally, another goal of the story was expressing the general value of experimentation and governance competition.  I think that existing work in the cause area of "improving institutional decisonmaking" too heavily focused on capturing the commanding heights of existing prestigious institutions and then implementing appropriate reforms "from the inside".  This is good, but it too should be complemented by the presence of more radical small-scale experimentation on the "outside" -- things like charter cities and experimental intentional communities -- which could test out wildly different concepts of ideal governance.

Below, I've selected some of the most relevant passages from my contest submission.  To get more of the sci-fi utopian flavor of what daily life would be like in the world I'm imagining (including two wonderful short stories written by my friend Holly, a year-by-year timeline, and more), visit the full page here.  Also, the Future of Life Institute would love it if you submitted feedback on my world and the other finalists -- how realistic do you find this scenario, how much would you enjoy living in the world I describe, and so forth.

Excerpts from my team's contest submissionIllustrating governance innovation, the Flash Crash War, the Delhi Accords & subsequent golden age.Artificial General Intelligence (AGI) has existed for at least five years but the world is not dystopian and humans are still alive! Given the risks of very high-powered AI systems, how has your world ensured that AGI has at least so far remained safe and controlled?

Ultimately, humanity was able to navigate the dangers of AGI development because the early use of AI to automate government services accidentally kicked off an “arms race” for improved governance technology and institution design. These reforms improved governments’ decision-making abilities, enabling them to recognize the threat posed by misalignment and coordinate to actually solve the problem, implementing the “Delhi Accords” between superpowers and making the Alignment Project civilization’s top priority.

In a sense, all this snowballed from a 2024 Chinese campaign to encourage local governments to automate administrative processes with AI. Most provinces adopted mild reforms akin to Estonia’s e-governance, but some experimented with using AI economic models to dynamically set certain tax rates, or using Elicit-like AI research-assistant tools to conduct cost-benefit analyses of policies, or combining AI with prediction markets. This goes better than expected, kickstarting a virtuous cycle:

  • Even weak AI has a natural synergy with many government functions, since it makes predicting / planning / administering things cheap to do accurately at scale.
  • Successful reforms are quickly imitated by competing regions (whether a neighboring city or a rival superpower) seeking similar economic growth benefits.
  • After adopting one powerful improvement to fundamental decisionmaking processes, it’s easier to adopt others (ie, maybe the new prediction market recommends switching the electoral college to a national-popular-vote with approval voting).

One thing leads to another, and soon most of the world is using a dazzling array of AI-assisted, prediction-market-informed, experimental institutions to govern a rapidly-transforming world.

 

The dynamics of an AI-filled world may depend a lot on how AI capability is distributed. In your world, is there one AI system that is substantially more powerful than all others, or a few such systems, or are there many top-tier AI systems of comparable capability?

Through the 2020s, AI capabilities diffused from experimental products at top research labs to customizable commercial applications much as they do today. Thus, new AI capabilities steadily advanced through different sectors of economy.

The 2030s brought increasing concern about the power of AI systems, including their military applications. Against a backdrop of rapidly improving governance and a transforming international situation, governments started rushing to nationalize most top research organizations, and some started to restrict supercomputer access. Unfortunately, this rush to monopolize AI technology still paid too little attention to the problem of alignment; new systems were deployed all the time without considering the big picture.

After 2038’s Flash Crash War, the world woke up to the looming dangers of AGI, leading to much more comprehensive consolidation. With the Delhi Accords, all top AI projects were merged into an internationally-coordinated Apollo-Program-style research effort on alignment and superintelligence. Proliferation of advanced AI research/experimentation outside this official channel is suppressed, semiconductor supply chains are controlled, etc. Fortunately, the world transitioned to this centralized a few years before truly superhuman AGI designs were discovered.

As of 2045, near-human and “narrowly superhuman” capabilities are made broadly available through API for companies and individuals to use; hardware and source code is kept secure. Some slightly-superhuman AGIs, with strict capacity limits, are being cautiously rolled out in crucial areas like medical research and further AI safety research. The most cutting-edge AI designs exist within highly secure moonshot labs for researching alignment.

 

How has your world avoided major arms races and wars?

Until 2038, geopolitics was heavily influenced by arms races, including the positive "governance arms race" described earlier. Unfortunately, militaries also rushed to deeply integrate AI. The USA & China came to the brink of conflict during the “Flash Crash War”, when several AI systems on both sides of the South China Sea responded to ambiguous rival military maneuvers by recommending that their own forces be deployed in a more aggressive posture. These signaling loops between rival AI systems lead to an unplanned, rapidly escalating cycle of counter-posturing, with forces being rapidly re-deployed, in threatening and sometimes bizarre ways. For about a day, both countries erroneously believed they were being invaded by the other, leading to intense panic and confusion until the diplomatic incident was defused by high-level talks.

Technically, the Flash Crash War was not caused by misalignment per se (rather, like the 2010 financial Flash Crash, by the rapid interaction of multiple complex automated systems). Nevertheless, it was a fire-alarm-like event which elevated "fixing the dangers of AI systems" to a pressing #1 concern among both world leaders and ordinary people.

Rather than the lukewarm, confused response to crises like Covid-19, the world's response was strong and well-directed thanks to the good-governance arms race. Prediction markets and AI-assisted policy analysts quickly zeroed in on the necessity of solving alignment. Adopted in 2040, the Delhi Accords began an era of intensive international cooperation to make AI safe. This put a stop to harmful military & AI-technology arms races.
 

In the US, EU, and China, how and where is national decision-making power held, and how has the advent of advanced AI changed that?

The wild success of China's local-governance experiments led to freer reign for provinces. Naturally, each province is very unique, but each now uses AI to automate basic government services, and advanced planning/evaluation assistants to architect new infrastructure and evaluate policy options.

The federal government's remaining responsibilities include foreign relations and coordinating national projects. The National People's Congress now mostly performs AI-assisted analysis of policies, while the Central Committee (now mostly provincial governors) has regained its role as the highest governing body.

In the United States, people still vote for representatives, but Congress debates and tweaks a basket of metrics rather than passing laws or budgets directly. This weighted index (life expectancy, social trust, GDP, etc) is used to create prediction markets where traders study whether a proposed law would help or hurt the index. Subject to a handful of basic limits (laws must be easy to understand, respect rights, etc), laws with positive forecasts are automatically passed.
This system has extensively refactored US government, creating both wealth and the wisdom needed to tackle alignment.

The EU has taken a cautious approach, but led in other areas:

  • Europe has created an advanced hybrid economy of "human-centered capitalism", putting an automated thumb on the scale of nearly every transaction to favor richer social connections and greater daily fulfillment.
  • Europe has also created the most accessible, modular ecosystem of AI/governance tech for adoption by other countries. Brazil, Indonesia, and others have benefited from incorporating some of the EU's open-source institutions.

 

What changes to the way countries govern the development, deployment and/or use of emerging technologies (including AI) played an important role in the development of your world?

After the world woke up to the dangers of powerful misaligned AI in 2038, nations realized that humanity is bound together by the pressing goal of averting extinction. Even if things go well, the far-future will be so strange and wonderful that the political concept of geopolitical “winners” and “losers” is impossible to apply.

This situation, like a Rawlsian veil of ignorance, motivated the superpowers to cooperate with 2040 Delhi Accords. Key provisions:

  • Nationalizing and merging top labs to create the Alignment Project.
  • Multi-pronged control of the “AI supply chain” (inspired by uranium & ICBM controls) to enforce nonproliferation of powerful AI — nationalizing semiconductor factories and supercomputer clusters, banning dangerous research, etc.
  • Securing potential attack vectors like nuclear command systems and viral synthesis technology.
  • API access and approval systems so people could still develop new applications & benefit from prosaic AI.
  • Respect for rights, plus caps on inequality and the pace of economic growth, to ensure equity and avoid geopolitical competition.

Although the Accords are an inspiring achievement, they are also provisional by design: they exist to help humanity solve the challenge of developing safe superintelligent machines. The Alignment Project takes a multilayered approach -- multiple research teams pursue different strategies and red-team each other, layering many alignment strategies (myopic oracle wrappers, adversarial AI pairs, human-values-trained reward functions, etc). With luck, these enable a “limited” superintelligence not far above human abilities, as a tool for further research to help humanity safely take the next step.

 

What is a new social institution that has played an important role in the development of your world?

New institutions have been as impactful over recent decades as near-human-level AI technology. Together, these trends have had a multiplicative effect — AI-assisted research makes evaluating potential reforms easier, and reforms enable society to more flexibly roll out new technologies and gracefully accommodate changes. Futarchy has been transformative for national governments; on the local scale, "affinity cities" and quadratic funding have been notable trends.

In the 2030s, the increasing fidelity of VR allows productive remote working even across international and language boundaries. Freed from needing to live where they work, young people choose places that cater to unique interests. Small towns seeking growth and investment advertise themselves as open to newcomers; communities (religious beliefs, hobbies like surfing, subcultures like heavy-metal fans, etc) select the most suitable town and use assurance contracts to subsidize a critical mass of early-adopters to move and create the new hub. This has turned previously indistinct towns to a flourishing cultural network.

Meanwhile, Quadratic Funding (like a hybrid of local budget and donation-matching system, usually funded by land value taxes) helps support community institutions like libraries, parks, and small businesses by rewarding small-dollar donations made by citizens.

The most radical expression of institutional experimentation can be found in the constellation of "charter cities" sprinkled across the world, predominantly in Latin America, Africa, and Southeast Asia. While affinity cities experiment with culture and lifestyle, cities like Prospera Honduras have attained partial legal sovereignty, giving them the ability to experiment with innovative regulatory systems much like China’s provinces.

 

What is a new non-AI technology that has played an important role in the development of your world?

Improved governance technology has helped societies to better navigate the “bulldozer vs vetocracy” axis of community decision-making processes. Using advanced coordination mechanisms like assurance contracts, and clever systems (like Glen Weyl’s “SALSA” proposal) for pricing externalities and public goods, it’s become easier for societies to flexibly make net-positive changes and fairly compensate anyone affected by downsides. This improved governance tech has made it easier to build lots of new infrastructure while minimizing disruption. Included in that new infrastructure is a LOT of new clean power.

Solar, geothermal, and fusion power provide most of humanity’s energy, and they do so at low prices thanks to scientific advances and economies of scale. Abundant energy enables all kinds of transformative conveniences:

  • Cheap desalinization changes the map, allowing farming and habitation of previously desolate desert areas. Whole downtown areas of desert cities can be covered with shade canopies and air-conditioned with power from nearby solar farms.
  • Carbon dioxide can be captured directly from the air at scale, making climate change a thing of the past.
  • Freed from the pressing need to economize on fuel, vehicles like airplanes, container ships, and self-driving cars can simply travel at higher speeds, getting people and goods to their destinations faster.
  • Indoor farming using artificial light becomes cheaper; instead of shipping fruit from the opposite hemisphere, people can enjoy local, fresh fruit year-round.

 

What’s been a notable trend in the way that people are finding fulfillment?

The world of 2045 is rich enough that people don’t have to work for a living — but it’s also one of the most exciting times in history, running a preposterously hot economy as the world is transformed by new technologies and new ways of organizing communities, so there’s a lot to do!

As a consequence, careers and hobbies exist on an unusual spectrum. On one end, people who want to be ambitious and help change the world can make their fortune by doing the all the pressing stuff that the world needs, like architecting new cities or designing next-generation fusion power plants.

With so much physical transformation unleashed, the world is heavily bottlenecked on logistics / commodities / construction. Teams of expert construction workers are literally flown around the world on private jets, using seamless translation to get up to speed with local planners and getting to work on what needs to be built using virtual-reality overlays of a construction site.

Most people don't want to hustle that much, and 2045's abundance means that increasing portions of the economy are devoted to just socializing and doing/creating fun stuff. Rather than tedious, now-automated jobs like "waiter" or "truck driver", many people get paid for essentially pursuing hobbies -- hosting social events of all kinds, entering competitions (like sailing or esports or describing hypothetical utopias), participating in local community governance, or using AI tools to make videos, art, games, & music.

Naturally, many people's lives are a mix of both worlds.

If you liked what you've read so far, remember to visit the official competition entry page to read the two day-in-the-life short stories and provide feedback!A Note of Caution

The goal of this worldbuilding competition was essentially to tell the most realistic possible story under a set of unrealistic constraints: that peace and prosperity will abound despite huge technological transformations and geopolitical shifts wrought by AI.

In my story, humanity lucks out and accidentally kick-starts a revolution in good governance via improved institution design – this in turn helps humanity make wise decisions and capably shepherd the safe creation of aligned AI.

But in the real world, I don’t think we’ll be so lucky.  Technical AI alignment, of course, is an incredibly difficult challenge – even for the cooperative, capable, utopian world I’ve imagined here, the odds might still be against them when it comes to designing “superintelligent” AI, on a short schedule, in a way that ends well for humanity.

Furthermore, while I think that a revolutionary improvement in governance institutions is indeed possible (it’s one of the things that makes me feel most hopeful about the future), in the real world I don’t think we can sit around and just wait for it to happen by itself.  Ideas like futarchy need support to persuade organizations, find winning use-cases, and scale up to have the necessary impact.

Nobody should hold up my story, or the other entries  in the FLI’s worldbuilding competition, as a reason to say “See, it’ll be fine – AI alignment will work itself out in the end, just like it says here!” 

Rather, my intent is to portray:

  • In 2045, an inspiring, utopian end state of prosperity, with humanity close to achieving a state of existential security.
  • From 2022-2044, my vision of what’s on the most-plausible critical path taking us from the civilization we live in today to the kind of civilization that can capably respond to the challenge of AI alignment, in a way that might be barely achievable if a lot of people put in a lot of effort.


Discuss

Why don't 4 lottery wins prove information from a higher power?

Новости LessWrong.com - 18 мая, 2022 - 18:10
Published on May 18, 2022 3:10 PM GMT

I would like to see a specific analysis of the errors, because I can only see for sure that frequency statistics are used, and not Bayesian. And so, the task: one popular Russian mathematician claims that since a woman won the lottery 4 times in a row, which is absolutely incredible, which is clear to any mathematician who, unlike mere mortals, sees the true essence of the universe, for example, that the chance of such a coincidence is one in a million billion billion, so this woman had information from the Christian god, because only he can break the laws of the highest of the sciences of mathematics, which is the language of god.



Discuss

Understanding Gato's Supervised Reinforcement Learning [Link Post]

Новости LessWrong.com - 18 мая, 2022 - 14:08
Published on May 18, 2022 11:08 AM GMT

The recent publication of Gato spurred a lot of discussion on wheter we may be witnessingth the first example of AGI. Regardless of this debate, Gato's makes use of recent development in reinforcement learning, that is using supervised learning on reinforcement learning trajectories by exploiting the ability of transformer architectures to proficiently handle sequential data.

Reading the comments it seems that this point created some confusion to readers not familiar with these techniques. Some time ago I wrote an introductory article to how transformers can be used in reinforcement learning which may be helpful to clarify some of these doubts: https://lorenzopieri.com/rl_transformers/ 



Discuss

Gato's Generalisation: Predictions and Experiments I'd Like to See

Новости LessWrong.com - 18 мая, 2022 - 10:15
Published on May 18, 2022 7:15 AM GMT

I'm deliberately inhabiting a devil's advocate mindset because that perspective seems to be missing from the conversations I've witnessed. My actual fully-reflective median takeaway might differ.

My covid has made writing difficult at the moment, and I haven't had the energy to gather citations or fully explain the detail for some of the assertions in this post.

I believe (for various reasons not detailed here) that qualitative advances in general agentic performance from artificial systems are likely this decade and next - I just don't think Gato represents progress in that direction. I'm not particularly surprised by anything in the Gato paper[1]. Naturally then, I'm against hyperbole around the capabilities demonstrated.

There is not enough information in the paper to say either way, but it may be the case that Gato represents a novel and cutting-edge distillation technique and if so, I would prefer if it were framed that way than as an increment towards generalisation ability of artificial agents!

The authors do a reasonable job of reporting their experiments (though I sense some motivated reasoning in the presentation of some results[2]), but the PR version of the story (at least as I have encountered it) departs from the mostly-honest paper itself[3].

I focus here almost entirely on sequential control, since that is the aspect of the discussion around the Gato paper which seems most clouded, while being at the centre of the purported generalisation, and in which I have the most interest and expertise.

Techniques and phenomena

We've known for many years that attention mechanisms (early 2010s) allow parameter sharing over sequences and over inputs of variable size/shape, that behavioural cloning (early 1990s?) often works with enough expert examples, that representation transfer (embeddings, early 2010s?) works, and that fine-tuning parameters (2000s? 2010s?) is faster than learning from scratch.

What should the key empirical takeaway be?

In my reading, if you subtract the already-known phenomena, and pass over the mostly-ignored distillation aspect, there is an important empirical finding in the Gato paper. I'm not sure if it's novel per se, but probably at this scale in this domain.

Namely, in our current systems, fine-tuning is typically faster when pretraining includes closely related tasks (e.g. DM Control Suite and Meta-World), slow when pretraining does not include closely related tasks (e.g. every control task without domain-relevant pretraining; including RGB stacking), and actively worse when pretraining includes distracting unrelated tasks (e.g. the single held-out Atari task, Atari boxing, is distracted by any pretraining in the reported experiments!).

Fine-tuning progress demonstrating benefit of closely related pretraining, and neutrality or harm of unrelated pretraining

Fine-tuning progress demonstrating benefit of closely related pretraining, and neutrality of unrelated pretraining

There is the additional novelty of goal-conditioning by context-prompting with a completed task, both during training and deployment (section 2 of the paper). Goal-conditioning is an interesting and relatively recent area of research. As with all such techniques I am aware of, this conditioning technique requires explicit presentation of a goal state which is structurally very similar to those seen in training. As briefly discussed in the paper, this could be applicable in the context of perceptual variation for closed-environment control (like some industrial robotics tasks) but this is fundamentally separate from general agency.

Comparison to other results regarding sequential control

Let's consider a comparison to the DQN papers published in NIPS and Nature in 2013.

Data efficiency

Note that the Gato control tasks were all learned by behavioural cloning on pretrained expert policies, each of which used a dedicated specific learning architecture, and were in many cases exposed to >10^8 environment steps for training each (depending on task; more on Atari; see appendix F of the paper).

Compare Nature DQN which had a quarter of the environment interaction experience per task on Atari.

Parameter count

Recall that the NIPS DQN paper learned Atari games without demonstrations (albeit per-game) using networks with less than a million parameters, and the final paper published in Nature used parameter counts little over a million. That's three orders of magnitude less than the larger Gato network (1.18B) and two-and-a-half less than the medium one used in ablations (364M), and this is before serious algorithmic and architectural optimisation investments started in deep RL - so we're not looking at a small network with surprising capabilities here, but rather a decently-sized network for the domains in question.

NIPS DQN[4]

type spec shape params 0 input 84x84 * 4 channels 84x84x4 - 1 conv 8x8 stride 4 * 16 channels 19x19x16 16*(8*8*4) = 2^12 2 conv 4x4 stride 2 * 32 channels 8x8x32 32*(4*4*16) = 2^13 3 ffn 256 256 256*(8*8*32) = 2^19 4 output <= 18 <= 18 <= 18*256 ~= 2^12

Nature DQN[5]

type spec shape params 0 input 84x84 * 4 channels 84x84x4 - 1 conv 8x8 stride 4 * 32 channels 19x19x32 32*(8*8*4) = 2^13 2 conv 4x4 stride 2 * 64 channels 8x8x64 64*(4*4*32) = 2^15 3 conv 3x3 stride 1 * 64 channels 6x6x64 64*(3*3*64) ~= 2^15 4 ffn 512 512 512*(6*6*64) ~= 2^20 5 output <= 18 <= 18 <= 18*512 ~= 2^13

(I assumed here they didn't use padding on the conv layers, and I ignored bias terms. The overal parameter count doesn't change much either way.)

Note also that the original DQN was able to achieve that performance while computing over only the 4 most recent frames, while the attention mechanism (and the recurrent mechanisms used in the expert policies) for the Gato paper will have access to much more history than that per decision step (perhaps ~50 frames for Gato, based on the context length of 1024 and image-tokenisation into 20 patches; see sections 2.3 and C.3).

Embeddings

In the Gato paper ResNet embeddings are also used for the image patches, but I can't find the details of these components specified - perhaps their parameters are in the low or fractional millions and either omitted from or absorbed into the parameter counts? But given that they're using residual connections, these image embeddings alone are presumably deeper than the DQN networks, if not larger.

Comparison conclusion

DQN used orders of magnitude fewer parameters, multiples less environment interaction experience, and orders of magnitude less compute per step, than Gato, on Atari.

Nature DQN Atari is a collection of neural networks. Gato's primary artefact is one (much much bigger) neural network (if you ignore the many individual neural networks used to learn the expert policies prior to behaviour cloning.)

Generalist or multi-purpose?

There are also learned embeddings for 'local position' (image-patch position and shape-relative embeddings for other input types). This is a sensible move and possibly the part of the paper which deserves to be called 'generalist' (as in, a general attention-based ML architecture).

In light of the above observations and comparisons, and some additional speculation detailed below, I consider the Gato artefacts ('agents') no more general than the collection of experts they were distilled from, though certainly they provide a multi-purpose model and demonstrate that distillation of multiple tasks into the same ML artefact is possible if you tag things appropriately and take care during training to avoid catastrophic forgetting.

Speculation

My best guess is that the domains and tasks trained have easily-detectable 'embedding signatures' via a combination of

  • explicit modality-dependent embedding (images vs text vs actions)
  • domain-dependent position encoding 'signatures' in the token-stream

Thus, I imagine as the Gato training algorithm learns it is doing something like

  1. learning good embeddings for different input modalities (as a consequence of diverse training data)
  2. extracting the 'embedding signatures' and internally tagging domains and tasks, for downstream 'switching'
  3. (most speculative) mostly routing domains and tasks to be learned/memorised 'separately'
  4. doing the usual attention-based sequence prediction thing on each domain/task

Of course it won't be as clean as this, but I bet a really good interpretabilitator would pull out some structure resembling the above pattern[6].

To the extent that Gato artefacts provide 'generalisation' capability, I expect it to be through transfer of good-quality representation embeddings learned over many related tasks, computed over the combination of the explicit token and position embeddings and the earlier layers of the attention network.

When domain- and task-specific feature representations are available, supervised fine-tuning on expert demonstrations should naturally be expected to rapidly and reliably recover good performance.

Any residual generalisation benefits from fine-tuning of cross-domain pretraining vs training-from-scratch may come from 'burn in' of the parameter space, effectively spending some training cycles getting it into a better 'initialisation' than the one that comes out of the box.

I do not expect any generalisation to come from anything resembling a general strategic, agentic, or planning core. I don't believe any of the Gato artefacts houses such a thing. For what it's worth, reading the paper, I get the impression that most of the authors also take this last position.

Experiments I'd like to see More hold-outs?

Purporting to demonstrate agentic generalisation demands more than four held-out tasks, and perhaps more than one held-out Atari task. Of course it costs money and time to run these experiments so they probably have not been tried yet.

Ideally there would also be a demonstration on actually out of distribution tasks. If there were any meaningful success here, my mind would be substantially changed, considering the reported negative generalisation performance on the single held-out Atari game.

Compute comparison between experts and distilled version

A comparison I would like to see would be between the Gato artefacts and the various expert policy models they were distilled from on

  • parameter count
  • runtime compute per step

If these comparisons are favourable, as noted above, it may be that Gato is best thought of as a novel technique for distilling policies and other expert behaviour, which would be an interesting result.

My best guess is that these comparisons are mostly not favourable to the Gato artefacts, but I'd be interested by counter evidence.

Ablation of image pretraining

For some tasks in the Gato paper, there was some residual benefit to pretraining on 'no control data' aka 'all but control data', compared to training from scratch. I imagine almost all of this benefit comes from pretraining the image embeddings. Note that the authors also speculate

agents in the DM Lab environment are fed images which, despite being simulated, are natural looking. Therefore, transfer from image captioning or visual grounded question answering tasks is possible.

This could be falsified by introducing new ablation categories for 'no image data' (include control but not image-based control) and 'no control or image data'[7]. In light of the above quote, I'm somewhat surprised not to find experiments like this in the paper.

Burn-in of parameter space

Initialisation of network weights is an area of experimentation and study; many approaches and heuristics exist, some of which will lend themselves to better learning rates than others.

A network at initialisation will have poorer weight/parameter settings than after some training(citation needed).

Here I'm referring to a fuzzy concept for which I do not have good technical language. Obviously parameters at initialisation are worse than after training (assuming training better than a random walk). But in addition to what we might think of as 'fine grained' learning and memorisation, there are also 'coarse grained' properties of initialisation which are poor, not specifically because they produce incorrect results, but because they are not yet in a good, readily-trainable region of parameter space.

For concreteness, consider properties like 'overall magnitudes' and 'diversity of weights' and the other heuristic properties which initialisation approaches are supposed to help with. The phenomenon of some amount of training cycles being spent 'burning in' the network is a common intuition in machine learning as practiced.

I don't have any experiment in mind for this one!

Perhaps ablating relative to amount of 'burn in' training on totally-unrelated data? But results of such experiments seem insufficiently-hypothesis-splitting to me: evidence for 'burn in' might also look like evidence for learning 'general transferrable agency'. I don't think the latter is what's happening, but the whole point of an experiment would be to disambiguate those hypotheses.

Uncovering implicit domain signatures

Evidence for 'domain signatures' or 'task signatures' would be the ability to 'easily' classify domain or task.

How many layers of the network's activations is required to classify domain/task by linear regression or small single-hidden-layer FFN?

  • taking padded single 'timestep frames' as input
  • taking n tokens as input
  • are the post-embedding raw tokens enough?
  • what about individual early layers of the attention network?
  • what about individual late layers of the attention network?
  • is there a network depth boundary where there's a phase transition?
  • how many principal components can we throw away and still classify?
  • how much distortion can we add and still classify? (e.g. squash activations to sign)
Explicit identification of domain- and task-tagging

Interpretability techniques applied to directions in the layer activations.

Can we uncover layer activation directions which correspond to domain- and task-tagging?

  • note that the tags and switching are likely to be rotated with respect to the canonical basis of the vector space of activations. See Features in Olah et al - Zoom In; Transformer attention networks in particular impose no real constraint on the orientation of features
    • so you might not find particular attention heads or weights dedicated to particular domains, but rather something more cryptic
  • can we locate domain- or task-specific highly-active directions?
    • what layers are they in?
    • is there a depth boundary where there's a phase transition?
Conclusion

There is a lot of great engineering reflected in the Gato paper, but the artefacts admit more potentially-revealing experiments than have yet been performed.

Depending on details not revealed in the paper, the techniques may represent a novel practical model-distillation technique.

I don't see evidence for the Gato artefacts being generalists (or agents, in a strict sense, for that matter), and my opinion is that claims to this effect are at best hyperbolic or confused. Readers (and writers) should take care to understand the multiple different possible connotations of words like 'general' and 'agent'.

  1. On the other hand, I do expect the discourse and framing to be such that the Gato paper is inevitably cited in any work from now on which does make progress in that direction (along with anything else which purports to). ↩︎

  2. Some examples I gathered on a subsequent reading:

    Section 4.1: (emphasis mine)

    For the most difficult task, called BossLevel, Gato scores 75%

    (footnote mentions that other tasks in fact scored lower)

    Section 5.2: (emphasis mine)

    In this section we want to answer the following question: Can our agent be used to solve a completely new task efficiently? For this reason, we held-out all data for four tasks from our pretraining set

    Section 5.5:

    The specialist Atari agent outperforms our generalist agent Gato, which achieved super-human performance on 23 games. It suggests that scaling Gato may result in even better performance.

    Appendix E:

    We evaluate agent every 100 learning steps. Each evaluation reports the average of 10 runs of a given checkpoint. The moving average of 5 such scores is computed (to gather 50 runs together). The final fine-tuning performance is defined as the maximum of these smoothed scores.

    ↩︎
  3. Based on professional experience, I react also (perhaps unreasonably) cynically to the density of phraseology using the 'generalist agent' bigram, and to the conflation of the techniques, training methods, architectures, and trained artefacts in the paper and public discussions under a single anthropomorphising moniker, 'Gato'.

    DeepMind publications usually read with much less of the marketing and hype aesthetic than this. ↩︎

  4. Mnih et al - Playing Atari with Deep Reinforcement Learning https://arxiv.org/pdf/1312.5602v1.pdf ↩︎

  5. Mnih et al - Human-level control through deep reinforcement learning https://www.datascienceassn.org/sites/default/files/Human-level Control Through Deep Reinforcement Learning.pdf ↩︎

  6. I suspect Transformer-style attention networks may be particularly well-suited to doing this sort of context-signature-tagging and switching-routing-separation and this is a large part of my model of how they work ↩︎

  7. Although see speculation on burn-in, which might confound such an experiment ↩︎



Discuss

I just watched the Open C3 Subcommittee Hearing on Unidentified Aerial Phenomena (UFOs). Here's a succinct summary and commentary + some background

Новости LessWrong.com - 18 мая, 2022 - 07:15
Published on May 18, 2022 4:15 AM GMT

Background on UFOs, Disclosure, and Rationality

There have been reports of various strange flying objects for a very very long time (read the paragraph with with "Boeotia". Note, a "hogshead" seems to be a type of barrel.). For me, it wasn't until quite recently that it became really unambiguously clear to me that something is really going on there. I'd recommend looking into the princeton/nimitz "tic-tac" incidents specifically. At least 6 navy staff on those boats have very publicly and extensively testified to having seen a very strange propulsion technology. I've come across no story as to how, or why any human faction would be keeping a propulsion technology like that secret, and out of deployment for so long.
(A half-baked theory though: Perhaps this propulsion tech could be used to make devilishly fast, non-interceptible ICBMs, the existence of which would make the world worse, and is therefore a technology that we should refuse to talk about for as long as possible?)

However, it's possible that it wasn't a propulsion technology, and that it's more of a plasma image projector used in consort with a radar spoofing technology, as futurist David Brin seems to confidently believe. In that case it's a lot easier to understand how the technology could have been kept hidden - it hasn't been mature for very long, and it has fairly limited applications, and it isn't salacious enough to leak about.
So this would be one of my dominant hypotheses, the tic-tacs really look like they're just this, to me. But there are a few contrary details in the account of commander Fravor and US officials swear up and down that they'd tell us if that were it.

So I kinda have to keep paying attention to this stuff.

But isn't it wildly implausible? How can a rationalist entertain the possibility that aliens would be like this, or behave like this? The econ/tech-eschatology of it doesn't make any sense!

I'm sympathetic to that. My intuition says that if aliens had reached us, their construction projects would have filled our night sky, they wouldn't be hiding from us, because whatever the value of hiding, the costs are greater. Robin Hanson talks extensively about the dynamics of economically plausible aliens and finds that it wouldn't generally support these sorts of observations.

But the observations seem to be disagreeing with my intuitions, so I seek a more detailed model.

Robin Hanson also talks extensively about the a less prevalent but still believable set of dynamics that we might expect to see in panspermia-sibling aliens who happen to have developed a sclerotic world government (a development which is not especially unlikely, and might happen to us) that could temporarily resist the laws of instrumental convergence towards "grabbyness".
That story would support these sorts of observations.

Regardless, if you are sure that it's definitely not aliens, you should be extremely interested in the possibility that humans, hence, appear to have created practical alcubierre drives.
I think that would warrant some discussion.

Summary and Commentary of the Proceedings of the Open Subcommittee

(The subcommittee hearing can be watched here)

There's a new US govt UFO investigation program: Airborne Object Identification and Management Synchronization Group, "Or, AIMSOG".

Congressman André Carson:

Today we will bring that organization out of the shadows. ... Unidentified Aerial Phenomena are a potential national security threat, and they need to be treated that way. For too long, the stigma associated with UAPs has gotten in the way of good intelligence analysis, pilots avoided reporting or were laughed at when they did, DoD officials relegated the issue to the back-room or swept it under the rug entirely, fearful of a skeptical national security community. Today, we know better. UAPs are unexplained, it's true, but they are real, they need to be investigated, and many threats they pose need to be mitigated.

  • Possibly new video of something reflective going past or being passed by a plane. Later in response to a clarifying question of Mr Himes, Mr Bray says yeah it's "probably moving very very slow".
  • They've figured out what that weird night vision footage of hovering triangles were! Just US drones producing a really weird camera lens effect in an IR lens being filmed by another camera. "We're now reasonably confident that these triangles correlate to unmanned aerial systems in the area. The triangular appearance, is a result of light passing through the night vision goggles, and then being recorded by an SLR camera"
  • (after faffing around trying and failing to get VLC to show a still of the flyby thing from that maybe new video. (It doesn't occur to anyone to try slowing down the playback :|||| like, should I be reading into the fact that there was this very basic technical puzzle in front of them and in all that time, not one of these guys thought of this))

    Mr Schiff: "Is this one of those situations where it was observed by the pilot and it was also recorded by the aircraft's instruments?", "Ah, we'll talk about the multisensor part, in a later session" (a closed one? :|| Why. It's possible this is just because talking about what the sensors saw would require talking about the sensors themselves, which is generally secret info.) "But in this case, we have, [gestures at the footage] at least that." :< neither confirming nor denying correlation from other sensors.
    I think I like Mr Schiff a lot. He is a live player. Regarding the hovering triangles, he questions this frankly surprising explanation optical artifacting, and asks whether they've reproduced that effect experimentally. Yes, apparently, it has been done.
  • Mr Wenstrup: 'have allies or adversaries reported similar sightings', "we should save that for a closed session" :[
  • Mr Gallagher asks them about the Malmstrom incident (I think this is it) where a UAP seemed to interfere some nuclear missiles. They say they haven't looked into it or heard of it. I'm as concerned as Gallagher is to hear that. They say they don't have the resources to follow up on that sort of story without an authoratative figure requesting it. Mr Gallagher says 'I don't claim to be an authoratative figure but for what it's worth I would like you to look into it'. Moultrie says "will do".
  • Mr Krishnamoorthi asks whether we have records of anything under the sea. Moultrie: "I think that would be more appropriately addressed in closed session, sir"
  • Mr Moultrie confirms that they do have processes "to ensure that we are not potentially reporting on something that may be a developmental platform or a US operational platform that is performing either testing or performing a mission".
    He states that there don't seem to have been any cases where they've accidentally reported a potential US technology. I don't place much stock in this, though. They wouldn't be able to do this job at all if they weren't willing to lie about whether something is a US technology, alas, we should expect them to lie about those cases, so maybe it doesn't ultimately mean very much that the government are consistently denying having any tech programs that could explain the observations.

I approach a model: It's conceivable that that what's actually going on here is, the story of Alien UFOs is useful for hiding novel technologies, but it's also been an impediment to national security, it's created a stigma around reporting, which means that if a rival nation actually had propulsion technologies like this, their radar crews might not report it. The US wants to maintain the cover of the AUFO story, but they also want to dissolve the stigma. I think this new group are going to do that exceptionally well.



Discuss

Workday Air Quality Measurements

Новости LessWrong.com - 18 мая, 2022 - 05:10
Published on May 18, 2022 2:10 AM GMT

I continue to be very excited about my new M2000 meter, and I brought it with me today to measure air quality during my commute and workday. Here's what I found:


full-size image

Time Activity CO2 (ppm) pm2.5 ug/m3 8:00 Left house 426 2.9 8:15 Davis station 468 69.9 8:24 Red line, new car 802 29.3 8:36 Kendall station, outside 477 11 8:40 Hallways, elevators, breakfast in cafe 563 2.8 9:18 Office, door open 475 1 10:00 Closed door 514 1 10:31 Open door 553 1 11:02 Team meeting, 14 people, ~15x30 1052 1 12:04 Hallways, elevator 686 1.6 12:10 Lunch in cafe 943 2.6 12:50 Hallways, elevator 763 2.1 12:57 Office, door open 537 1 2:02 Closed door 539 1 2:31 Two people 599 1 2:54 One person, open door 587 1 3:01 Closed door, two people 598 1 3:31 Open door, zero people 538 1 4:04 Closed door, two people 591 1 4:38 Kendall station 487 36.6 4:44 Red line, old car 953 21.6 4:56 Davis station 537 134.5 4:58 Outside 445 2.25

I spent most of my workday in my office, with the door open. I closed the door for meetings, either virtual or in person. The "zero people" at 3:31 is me going on a walking 1:1: with a report and leaving my meter behind at my desk.

Currently I wear a mask on the subway (and stations) and at work when I'm not in my office or a conference room with a small number of people. Based on these readings, it would be safe from a covid perspective to remove my mask in the subway station, but given the high level of particulate pollution I might as well leave it on.

(At this point I'm not wearing a mask because I think it would be directly harmful to myself or my family to get covid, but instead because the steps we would need to take to reduce the risk of infecting others once we knew we were infected would be very inconvenient.)



Discuss

Feature request: draft comments

Новости LessWrong.com - 18 мая, 2022 - 00:21
Published on May 17, 2022 9:21 PM GMT

When thinking about somebody’s post, I often benefit from writing out my thoughts. Usually, I do that in a comment box, because that’s convenient.

Often, I’ll realize that my first thought is wrong, hit on something I want to read before posting, not have time to finish my thought, or feel growing uncertainty about whether it’s worthwhile investing the time to finish my comment. I don’t want to keep track of a bunch of half-formed thoughts on, say, Google Drive. So I delete the comment instead.

It would be nice if I could easily make and save a bunch of hidden comments on the post itself. They’d be essentially “draft comments.”



Discuss

Popular education in Sweden: much more than you wanted to know

Новости LessWrong.com - 17 мая, 2022 - 23:07
Published on May 17, 2022 8:07 PM GMT

Growing up on the Swedish seaside, I had a five-minute walk to four open learning facilities – not counting the library and the youth center. It was very Christopher Alexander.

One of the premises was an abandoned church that my friends and I used as a recording studio; we'd renovated it ourselves with funding from a study association. There we played distorted pop. In another, I learned French from an émigré of Montpellier. We arranged public lectures – once, to our great surprise, we even managed to book then general secretary of the United Nations Ban-Ki Moon for a lecture in Uppsala. I analyzed Soviet cinema with a group of whom an unsettling number sang Sång för Stalin before the screenings.  

Since leaving Sweden, I have realized that not everyone grows up like this. And I miss it. In fact, if the whole of Sweden was about to burn down and I could only save one thing, I might grab just folkbildningsrörelsen.

Folkbildningsrörelsen: that is the name we have for this movement of self-organized study groups, resource centers, maker spaces, public lectures, and free retreats for personal development. 

These types of things exist in other countries too – but not at the same scale. Or even close.

To get a sense of how comprehensive folkbildningsrörelsen is, it helps to remember that Sweden has a population roughly comparable to New York City. If NYC had as many free resource centers per inhabitant as the municipality where I grew up, Manhattan would look like this:

I was going to do all of New York but my hand started hurting from making all these dots, so I only managed the tip of Manhattan.

At every other intersection, there would be a few rooms where you could go in and get some money to buy literature or access tools you needed. (In practice, the resource centers in cities tend to be lumped together in larger units, but the map still captures a lived reality for the 7.5 percent of Sweden's population who regularly take part in study associations.) 

Experientially, the spaces I have been part of have felt more like niche internet forums than schools. There were plenty of trolls, witches, and freaks – but we were also able to sustain a depth of conversation which was out of scope at school. When I entered university, seminars often felt like play-acting in comparison. In our often quite dilapidated buildings (as in internet communities), we hadn’t thought about what we were doing as learning

We were just obsessing about things.

How did this all come about?

In the 19th century, when these houses and the financing that enables them began to be built out, the main impetus came from the German Bildung tradition. 

Bildung etymologically refers to shaping yourself in the image (das Bild) of God. God in this context should be imagined as a highly self-possessed spectral being – in control of its emotions, with mind and heart in harmony, and willing to take individual moral responsibility. Think Bertrand Russell but less atheist, and sitting on a cloud.

This is the look.

In its original formulation, Bildung had a somewhat bourgeois flavor. It smelled of tweed and leather elbow patches. But in the early 1800s, thinkers such as Johann Heinrich Pestalozzi, N.F.S. Grundtvig, and Johann Friedrich Herbart figured out how to sell Bildung to farmers and day laborers – a folk Bildung, or folkbildning in Swedish. This was the tradition that took root in Sweden: the popular movement to shape yourself in the image of Bertrand Russell.

The English language version of folkbildning’s Wikipedia page refers to it as popular education. This translation is not entirely correct. The term "popular education" has a strong political connotation – the Wikipedia page talks about "class, political struggle, and social transformation" – which does not map entirely to the present Scandinavian reality. Though the study associations grew out of popular movements (the free church movement, followed by the temperance movement and the labor movement) – these spanned the political spectrum. And the learning infrastructure they spawned, as we will see, rapidly outgrew their political aims.

Building intellectual retreats for farmers

Since the 17th century, there have been popular educational movements in Sweden. Perhaps most interesting among these were the so-called Readers, who arranged reading lessons for peasants and opened secret libraries. The state, which had its own compulsory reading education, persecuted the Readers, fearful of the social unrest that might result from free reading.

In the 19th century, the popular education movement started to grow into a significant societal force. This began with the creation of so-called folk high schools (folkhögskolor). These first emerged in Denmark, in Ryslinge, where Christen Kold in 1851 started a school based on N.F.S. Grundtvig's idea of an ungraded, discussion-focused institution for higher education, aimed at the lower classes.

Folk high schools were located in scenic areas - not so much to be romantic retreats for city dwellers but to be close to the farmers who were their main clientele. In The Nordic Secret, Björkman and Andersen argue that folk high schools were retreats for ego development along lines similar to Robert Kegan's. It was about creating the conditions for people who had lived in simple small-scale communities to develop the knowledge and psychological complexity required to navigate modern society. Much emphasis was placed on discussions, practical skills and simulations.

The first Swedish folk high schools, Hvilan, Önnestads, and Herrestad, started seventeen years later, in 1868, seemingly without contact with the Danish movement.

They arranged role-playing events where workers and farmers played out committee meetings and other arcane parts of the political process. This meant that once they got the vote and started sweeping into office, the worker representatives out-maneuvered the representatives from the upper classes, to the great surprise of many who had argued against democracy on the grounds that it would lead to a flood of unwashed plebeians. The secretaries in the government office, who were in the habit of grading political representatives for their professionalism, left good marks for the early workers' representatives.

At their peak, 10 percent of young adults in rural areas choose to attend folk high schools. Björkman and Andersen's thesis is that this created a critical mass, well distributed in the population, that had the intellectual and emotional tools needed to effectively navigate a complex society. This, in turn, would explain the rapid transition that the Nordic countries made, from being the poorest in Europe in the 1850s to being the happiest, most equal, and nearly richest societies in the world eighty years later. I think that is overplaying the importance of the folk high school – but it does gesture at the transformative impact that popular education had on large swaths of the population.

And it was only just beginning.

How folkbildning went viral

Folk high schools were an important part of transforming public education into a societal force. Folkbildning was spreading.

Then, in 1902, Oscar Olsson, a secondary-school teacher who looked like he was living through the final stages of a pandemic lockdown, figured out how to make it go viral.

Olsson had returned from a trip to the United States where he had observed the success of the Chautauqua movement, an educational spectacle with speakers, showmen, and preachers, which Theodore Roosevelt, quite aptly, called “the most American thing in America”. Now Olsson was trying to figure out how to bring these ideas to his Good Templar lodge in Lund, to help his fellow Good Templars spread temperance.

What he came up with was a Scandinavian, minimalist version of Chautauqua, which he called a study circle. The study circle, Olsson envisaged, would be made up of equals and elect a leader from among its members. It would take literature as its starting point, and help its members acquire knowledge in the course of free conversation. It was, as all good memes are, a very simple idea. And it was cheap. The members (numbering between 5 and 20) could, if necessary, meet at home and would choose their own study material. That created an economically viable form of education for the working class. 

And it was made even more viable three years later, when the Riksdag, Sweden's parliament, voted to give grants for the purchase of books, on the condition that the books were made available to the general public1.

Another factor behind the success of study circles was their focus on communal self-improvement. Study circles were a child of the temperance movement – a movement that neither sought collective power, like the unions, nor self-improvement for the individual, but rather encouraged people to improve to serve their fellow human beings. This focus on communal self-improvement seems to have provided momentum to the movement. It also helped foster social capital formation, creating dense high-trust networks.

The spread of study circles

The study circle model spread rapidly from the Good Templar Order through the popular movements, filling up a public depot of books – which became the foundation of Sweden's public library system. 

Since the members were working-class or small farmers, the study circles, like the Folk high schools, played an important role in preparing them for their growing political power in the 20th century. Farmhands, miners, and dock workers were studying not only political and social issues but accounting, law, literature, math – everything they would need on the day of reckoning when they would run the country. They worked weekends building houses to study in. They formed national study associations to facilitate the supply of funding. They were dead serious about becoming small Bertrand Russells.

It was not an education system. Rather, it was an attempt to unleash what I have called the learning system. Instead of interventions aimed at controlling what people learn – which is how we can think about traditional education – folkbildningsrörelsen provided people with the resources they needed to learn on their own. The movements created the conditions for an ecosystem to emerge.

In 1950, 30,000 study circles were active. One for every 200th person. Several of the architects of the modern Swedish state – such as Gunnar Sträng and Torsten Nilsson – had been raised in this ecosystem of popular education.

Twenty years later, in 1970, the number of study circles had grown to 300,000 – counting 10 percent of the population as active members. 

Since then it has fluctuated around the same absolute numbers, today counting 7.5 percent of the population as active members.

By the time the movement reached its peak, the original political aspirations had somewhat waned. The waning was partially a result of the funding: to receive state funding, the associations had to give up political control of the curriculum. Both the state and the associations battled to control the curriculum, but none turned out to be strong enough to force the other, so in a sort of Westphalian peace, the control was ceded to the learners. 

And, as with anything that gets decentralized, variance and variety increased.

Retired women started forming study circles to throw pottery together. There were study circles for people who were anxious about the H day, in 1967, when Sweden would switch from driving on the left side of the road to the right. White-collar workers, envious of blue-collar associations, started their own – though they favored classroom studies over discussion groups, and mostly studied accounting and how to actually run an association. Universities formed spin-offs in the form of associations to broadcast public lectures and attract students to language schools. Many study circles also, on closer inspection, turned out to be bands that wanted access to cheap rehearsal spaces.

How study circles works in practice

To start a study circle today you:

join a study association, 

get instructions on how to report your studies, 

submit forms to document how many participants have attended and how many hours have been spent on the studies, and

receive financial support in proportion to your efforts - about $2 per man hour - which is often invested in renting rooms but can also be spent on educational materials, tools, or services.

Until I moved from Sweden, I administered three study circles. The paperwork took about 30 minutes a year. In two cases, the study circles were book clubs. Incorporating as study circles gave us access to meeting rooms. And sometimes, when we invited researchers to discuss their ideas with them, we could use study funds to pay for their train tickets. When we wanted to learn something more practical, we used the funds to buy services - such as a recording engineer, who could sit with us when we recorded music and give us hands-on guidance.

There is, of course, financial waste in the system, and one can debate the morality of funding book clubs with taxpayers' money. But on the whole, the 40 or so study circles I have observed got more learning per penny than the compulsory schools I have worked at. 

I also can't imagine living in a city without access to basement rooms where you can go down and find people all day engaged in strange discussions and projects. When I find myself in cities where the only places to sit down and talk are cafés and bars, I get intense claustrophobia.

Is popular education a viable alternative to public schooling?

A good educational system should have three purposes: it should provide all who want to learn with access to available resources at any time in their lives; empower all who want to share what they know to find those who want to learn it from them; and, finally, furnish all who want to present an issue to the public with the opportunity to make their challenge known. 

– Ivan Illich

In 1971, when Ivan Illich published Deschooling Society, his scathing critique of schooling and call for voluntary learning webs, the Swedish study circles had already become something akin to what he proposed. It was “a set of services that did not coerce anyone to learn but provided all who wanted to with access to available resources and interested peers”. The study circles “helped those who wished to share what they knew to find those who wanted to learn it from them”. They were voluntary, and free from government control.

Popular education is and should be free [from government control] and voluntary. This free and voluntary popular educational work enables all to seek knowledge on the basis of their own experience, preferences and learning style, without limitation from demands for results, and without mechanisms of exclusion. The approach permits dialogue, involvement and questioning, without a preconceived framework. 

– Swedish Government Bill 1997/98:115:5

Ivan Illich’s ideas of a deschooled society are easy to dismiss as the ravings of a defrocked Catholic priest determined to restore Christianity to its prime (which was the first century AD, in case you wonder). But does Sweden’s experience indicate that his ideas are feasible? Would a non-coercive learning system work as the main educational infrastructure in modern society?

Looking at the current state of popular education in Sweden, the movement is more a vehicle of self-cultivation, than a balanced reproduction of the knowledge we need to sustain our civilization. The most common subjects are related to arts and crafts – people play in bands, learn how to restore houses, knit, and garden. Foreign languages are also popular. More advanced studies in STEM subjects are notably absent (though traditional school subjects at the high school level, including STEM, are popular). 

Two theories of why the movement became dominated by hobbyists

The first generations of the movement were more focused on serious skill-building. Why did that change? One theory is that the early members were outliers that eventually were outnumbered by hobbyists as the movement scaled. It was an inevitable, eternal September.

A second theory as to why popular education became a hobby project after 1950 is this. The postwar boom was the period when school ultimately cemented its role as the only acceptable way to signal employability (Gustav Möller, who lost the bid for the party leadership of the Social Democrats to Tage Erlander in 1946, was probably the last major political figure to argue social services like education should be arranged through voluntary associations, not directly through the state). With Erlander and Palme, the state minister and his protégé, there was a push for the “strong state” (det starka samhället) – which included more schooling as well as increased state control over higher education. Credentials, especially engineering degrees, became exalted. And in a society where only certified learning is considered legitimate, self-directed learning becomes consumption.

There is a measure of truth to both of these theories. Most of us are more attracted to self-realization than to bookkeeping – and you have to be doubly insane if you study bookkeeping without the promise of a career. Or at least a worker-run utopia.

Other limitations of the model

There are a number of other limitations to the Swedish model, beyond not being strong in STEM fields. It has so far mostly been limited to people over 13. Giving young children access seems to me a missed opportunity. Merging the popular education movement with homeschooling is a promising path to pursue.

Another area where there is room for improvement is the pedagogy. The popular education movement has a strong emphasis on discussion-based learning, which has many merits but is now often used in contexts where it is a bad fit. Several of the most effective learning methods (such as simulations and case studies, deliberate practice, spaced repetition, apprenticeships, and one-to-one tutoring) are rarely used. Of course, the same thing can be said of schools.

And in defense of Illich's idea of keeping learning voluntary, people of all classes were learning advanced skills before mandatory education. The Readers spread literacy among farmers2, workers self-studied accounting and law... hell, even the cathedrals were built by self-learners.

Bottom-up growth of learning infrastructure

Throughout history, we have seen new learning services spring up organically as the demand for more advanced skills increase - aristocratic tutors in the Roman Empire, Imperial examinations in China, the more structured apprenticeships that spread during the Renaissance, the guilds of the middle ages, grammar schools and cheaply printed reading primers after Gutenberg. And the broad willingness of workers and farmers to labor on weekends and evenings to build lodges where they could study suggests that the Swedish population was in full swing building a more advanced education system during the first half of the 20th century. Then they stopped. They were outcompeted for time, resources, and legitimacy by state schools.

Had they not stopped, had they kept building and experimenting, what would this popular alternative have looked like today?

It is naturally impossible to say where this counterfactual history would have ended up. But if we can draw any conclusions from the Swedish experience, it would have been somewhere much stranger and more fun than here.

And we can still find out.

Acknowledgments

This piece has been shaped by the comments of several people, here sorted in the order of how much work I had to put in to please them: Johanna Wiberg, tracingwoodgrains, Matt Smith, and Justis Mills at LessWrong.

References

This essay is a summary of several books and articles, some of which are in Swedish. I have also relied on my memory and done some surveying of the homepages of the major study assocations.



Discuss

On saving one's world

Новости LessWrong.com - 17 мая, 2022 - 22:54
Published on May 17, 2022 7:53 PM GMT

If the world is likeliest to be saved by sober scholarship, then let us be sober scholars in the face of danger.

If the world is likeliest to be saved by playful intellectual exploration, then let us be playful in the face of danger.

Strategic, certainly; aware of our situation, of course; but let us not throw away the one mental mode that can actually save us, if that's in fact our situation.

If the world is likeliest to be saved by honest, trustworthy, and high-integrity groups, who by virtue of their trustworthiness can much more effectively collaborate and much more quickly share updates; then let us be trustworthy. What is the path to good outcomes otherwise?

CFAR has a notion of "flailing". Alone on a desert island, if you injure yourself, you're likelier to think fast about how to solve the problem. Whereas injuring yourself around friends, you're more likely to "flail": lean into things that demonstrate your pain/trouble to others.

To my eye, a lot of proposals that we set aside sober scholarship, or playful intellectual exploration, or ethical integrity, look like flailing. I don't see an argument that this setting-aside actually chains forward into good outcomes; it seems performative to me, like hoping that if our reaction "feels extreme" enough, some authority somewhere will take notice and come to the rescue.

Who is that authority?

If you have a coherent model of this, we can talk about it and figure out if that's really the best strategy for eliciting their aid.

But if no one comes to mind, consider the possibility that you're executing a social instinct that's adaptive to threats like tigers and broken legs, but maladaptive to threats like Unfriendly AI.

If you feel scared about something, I generally think it's good to be honest about that fact and discuss it soberly, rather than hiding it. I don't think this is incompatible with rigorous scholarship or intellectual play.

But I would clearly distinguish "being honest about your world-models and feelings, because honesty is legitimately a good idea" from "making it your main strategy to do whatever action sequence feels emotionally resonant with the problem".

An "extreme" key doesn't necessarily open an "extreme" lock. A dire-sounding key doesn't necessarily open a dire-feeling lock. A fearful or angry key doesn't necessarily open a lock that makes you want to express fear or anger.

Rather, the lock's exact physical properties determine which exact key (or set of keys) opens it, and we need to investigate the physical world in order to find the right key.



Discuss

Proposal: Twitter dislike button

Новости LessWrong.com - 17 мая, 2022 - 22:40
Published on May 17, 2022 7:40 PM GMT

The popular story of Twitter’s role in the ruin of civilization is that it is a runaway trash fire of reciprocal anger and offense, where otherwise nice people are possessed by overwhelming outrages, and drawn into throwing their own energy behind creating the vilest and most vindictive responses to what they see, turning away from reason and hurting others in turn, and so the place continues.

I’m not sure how much of Twitter activity this accounts for (apparently Michael Nielsen enjoys an entirely different place, and my experience seems pretty nice too). But I think there’s a real pattern of this kind, which makes game theoretic sense, and goes something like this:

  1. People say things
  2. People read these things
  3. If anything seems objectionable to any of these people, then they repost those things with commentary, and everyone else reads them extra.
  4. In the next round, people (or the ones who who get attention) say objectionable things (that they expect will get attention), about objectionable things (that they have their attention on from the last round)
  5. etc.

To lay out the effects of all this more clearly:

  1. People disproportionately read things they don’t like, which is presumably bad for them
  2. People get the visceral sense that others are disproportionately writing things they don’t like, which is misleading, and not in a helpful-for-public-friendship way
  3. Things people don’t like get extra space in the public conversation
  4. People who tend to write things that others don’t like get extra power and attention instead of less
  5. Writing things other people don’t like is incentivized (if you want attention, writing things other people don’t like is probably somewhat better than writing things people do like, and way better than writing things they don’t feel strongly about).

Supposing something like this model is true, and bad, it seems to me that there is a really simple solution: add a dislike button.

That is, what if when a person sees a thing they don’t like, instead of broadcasting it to others, they register their disapproval by quietly clicking a different button next to the heart, and then Twitter shows it to other people less instead of more? You can still retweet it if you especially want other people to see it more, but adding attention wouldn’t be the default disapproval vote.

This is not an original idea, and the other major websites that do it have not, to my knowledge, been run out of business by a dearth of disagreement. I think they are also not so much known for the above dynamic.

I posit that a Twitter downvote button would be great. What am I missing?



Discuss

Страницы

Подписка на LessWrong на русском сбор новостей