Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 43 минуты 20 секунд назад

Dark Mode on LessWrong

10 мая, 2022 - 04:21
Published on May 10, 2022 1:21 AM GMT

One of the more commonly requested features on LessWrong is a dark mode. Well, now we have it! To activate, hover over the user-menu in the top right, open the Themes submenu, and click on Dark Mode there.

Notes:

  • This feature is beta, by which I mean there may be aesthetic problems, especially in less-used areas of the site. Please give feedback in the comments here. Hopefully there is no black-on-black text anywhere, but this is the sort of thing that could happen, so if you do come across any then please report it.
  • You need to be logged in to pick a theme, for UI reasons (the theme-select option is in the user menu because there wasn't a good place to put it). If you log in, switch to dark mode, and then log back out, it will remember your choice, though, so you don't have to stay logged in if you don't want.
  • Posts that use custom colors will mostly have their colors overridden to defaults, to prevent black-on-black text. We do have a way to provide dark-mode overrides for specific custom colors which we've used on a few specific posts. If there are important historical posts with black-on-black problems, or which use custom colors in an important way that didn't come through, please let me know in the comments here and I'll try to fix it.
  • For technical reasons, we don't inherit the operating system dark-mode preference. Let us know in the comments if this is important to you and we might prioritize it.


Discuss

AI Alignment YouTube Playlists

10 мая, 2022 - 02:06
Published on May 9, 2022 9:33 PM GMT

I created two AI Alignment playlists on Youtube. One that is slide-heavy and the other is not. I separated them into two playlists for two reasons.

  1. It’s useful to separate for a dataset I am working on.
  2. Media is easier to consume when you don’t have to pay attention to the slides and pictures someone is describing.

Not slide-heavy (currently 216 videos): https://youtube.com/playlist?list=PLTYHZYmxohXp0xvVJmMmpT_eFJovlzn0l 

Slide-heavy (currently 366 videos): https://youtube.com/playlist?list=PLTYHZYmxohXpn5uf8JZ2OouB1PsDJAk-x

If you would like to contribute and add more videos to the playlists or create new Alignment-relevant playlists, let me know!

If you like access to the audio and youtube auto-generated subs in .txt format, I have stored them here: https://drive.google.com/drive/folders/1qVo4TyHKrsJvbJ3UrIOLW45j_7_wwnbZ?usp=sharing 

I've batched up the files into buckets of 90-ish hours (except for the final bucket which is less) since I plan on loading them into otter.ai and that website only accepts 100 hours per user (per month). Additionally, if you would like to help load some of the audio files in your own otter.ai account, please let me know! I want to create transcripts of the audio files and add them to a dataset very soon.



Discuss

Conditions for mathematical equivalence of Stochastic Gradient Descent and Natural Selection

10 мая, 2022 - 00:38
Published on May 9, 2022 9:38 PM GMT

Many thanks to Peter Barnett, my alpha interlocutor for the first version of the proof presented, and draft reader.

Analogies are often drawn between natural selection and gradient descent (and other training procedures for parametric algorithms). It is important to understand to what extent these are useful and applicable analogies.

Here, under some modest (but ultimately approximating) simplifying assumptions, natural selection is found to be mathematically equivalent to an implementation of stochastic gradient descent.

The simplifying assumptions are presented first, followed by a proof of equivalence. Finally, a first attempt is made to consider the effect of relaxing each of these assumptions and what departure it causes from the equivalence, and an alternative set of assumptions which retrieves a similar equivalence is presented.

Summary of simplifying assumptions

It is essential to understand that the equivalence rests on some simplifying assumptions, none of which is wholly true in real natural selection.

  1. Fixed 'fitness function' or objective function mapping genome to continuous 'fitness score'
  2. continuous fixed-dimensional genome
  3. radially symmetric mutation probability density
  4. limit case to infinitessimal mutation
  5. a degenerate population of 1 or 2
  6. no recombination or horizontal transfer
Proof Setup and assumptions

Let us make some substantial modelling simplifications, while retaining the spirit of natural selection, to yield an 'annealing style' selection process.

We have continuous genome xt∈Rn.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} with fixed fitness/objective function f:Rn→R

Mutations are distributed according to probability density Pm:Rn→[0,1] which is radially symmetric

  • so ∀ϵ∈Rn.Pm(ϵ)=g(|ϵ|) for some g:R+→[0,1] (a density, but not a probability density)
  • a mutation in one direction is just as likely as a mutation in another direction

We also consider the case of mutations very small relative to the genome, tending to infinitessimal.

Selection is stochastic, but monotonically determined by the fitness differential, according to selection function Ps:R→[0,1]

  • so the probability of x′ being selected over x is Ps(f(x′)−f(x))
  • Ps is a monotonically nondecreasing function: a greater 'fitness score' differential can not reduce chances of selection
    • i.e. \delta \implies P_s(\Delta) \ge P_s(\delta)">∀Δ,δ∈R.Δ>δ⟹Ps(Δ)≥Ps(δ)
  • a good model for this is something like a softmax, Boltzmann, or logistic function[1], i.e. a normalised ratio of exponentials, but this is not essential for the proof

The population alternates between a single parent and its two offspring, one of which is selected to become the next generation's parent.

  • selection according to Ps abstracts any mechanism which results in becoming the ancestor of future generations
    • not dying first
    • reproducing sufficiently successfully to 'win'
    • capturing/converting resources more efficiently
  • in this first setup, all offspring are uniparental and no genes are mixed by any means other than mutation

Theorem

Now consider a binary fission from xt, yielding one perfect clone and one mutant clone x′t where mutation (xt+1−xt)∼Pm.

Define the next 'step', xt+1 as whatever mutant offspring x′t eventually happens to be successfully selected vs a perfect clone, according to the selection function on their fitness differential. (If the perfect clone gets selected, it may take many intermediate generations to constitute a 'step' here.) Denote by αs the resulting normalisation constant over mutations.

So the distribution over xt+1 given xt is

P(xt+1|xt)=αsPm(xt+1−xt)Ps(f(xt+1)−f(x))

Call the mutation ϵ=xt+1−xt. Then we find

P(ϵ|xt)=αsPm(ϵ)Ps(f(xt+ϵ)−f(xt))≃αsPm(ϵ)Ps(ϵ⋅∇f(xt))

by considering the directional derivative of f along ϵ at xt and the limit as ϵ→0[2]. (Prior to this infinitessimal limit, we have instead the empirical approximation to the directional derivative.)

Now characterising ϵ by length rϵ=|ϵ| and angle-from-gradient θϵ=∠(∇f(xt),ϵ)

P(ϵ|xt)≃αsg(rϵ)Ps(rϵ|∇f(xt)|cosθϵ)

At this point it is clear that our step procedure depends, stochastically, on how closely the direction of the mutations match the fitness function's gradient.

By inspecting the expected value of the step direction, θϵ, we can make a more precise claim

E[θϵ|xt]=∫P(ϵ|xt)θdϵ≃∫αsg(rϵ)Ps(rϵ|∇f(xt)|cosθϵ)θϵdϵ

and finally, by noticing the integral of an odd function[3] in θ

E[θϵ|xt]≃0

Thus the update between steps, ϵ, is a stochastic realisation of a variable whose orientation is, in expectation, exactly the same as that of the gradient ∇f(xt) of the fitness function.

By similar inspection of E[rϵ|xt] we can see that it is a monotonic function of |∇f(xt)|, depending on the particulars of Pm and Ps, which together provide a gradient-dependent 'learning rate'.

So natural selection in this form really is nothing but an implementation of unbiased stochastic gradient descent!

Discussion of simplifying assumptions

To what extent are the simplifying assumptions realistic? What happens to the equivalence when we relax any of the assumptions?

Fixed 'fitness function'

In real natural selection, the interaction between a changing environment and a dynamic distribution of organisms collaborating and competing leads to a time- and location-dependent fitness function.

Variable fitness functions can lead to interesting situations like evolutionarily stable equilibria with mixtures of genotypes, or time- or space-cyclic fitness functions, or (locally) divergent fitness functions, among other phenomena.

Such a nonstationary fitness function is comparable to the use of techniques like self-play in RL, especially in conjunction with Population-Based Training, but is less comparable to vanilla SGD.

As such it may be appropriate to think of real natural selection as performing something locally equivalent to SGD but globally more like self-play PBT.

Continuous fixed-dimensional genome and radially-symmetric mutation probability density

Moving from a continuous to a discrete genome means that the notion of a gradient is no longer defined in the same way, but we can still talk about empirical approximate gradients and differences.

The mechanisms which introduce mutations in real natural selection are certainly symmetrical in certain ways, but probably not in any way which straightforwardly maps to radial symmetry in a fixed-dimensional vector space.

Without radial symmetry, much of the mathematics goes through similarly, but instead of an unbiased estimate of the gradient direction, it is biased by the mutation sampling. As such, we might think of real natural selection as performing a biased stochastic gradient descent.

A comparison may be made to regularisation techniques (depending on whether they are construed as part of the training procedure or part of the objective function), or to the many techniques exploiting bias-variance tradeoffs in sampling-based gradient-estimation for RL, though these tend to be deliberately chosen with variance-reduction in mind, while natural selection may not exhibit such preferences.

Limit case to infinitessimal mutation

In reality, mutations are not infinitessimal, but in practice very small relative to the genome. If we do not take the limit, instead of an exact directional derivative, we find an empirical-approximate directional derivative, yielding empirical-approximate stochastic gradient descent.

This means that in real natural selection, the implied 'step size' or 'learning rate' is coupled with the particulars of the selection strength, the variance of the stochastic gradient, and the degree of empirical approximation applied. In contrast, stochastic gradient descent per se need not couple these factors together.

A degenerate population of 1 or 2

If we expand the population to arbitrary size, it is possible to retrieve the equivalence with additional assumptions.

Instead of a parent individual and cloned and mutated offpsring individuals, considering parent and offspring populations, the same reasoning and proof is immediately applicable if we assume that mutations arise sufficiently rarely to be either fixed or lost before the next mutation arises. In this case Ps, the probability of selection, becomes the probability of fixation.

Of course this is not true for real natural selection.

If instead we allow for multiple contemporary mutant populations, an identical treatment can not be applied.

No recombination or horizontal transfer

One of the most fascinating and mathematically complicating aspects of real natural selection is multiple heredity of genome elements, whether via horizontal transfer or sexual recombination.

The preceding proof of equivalence for natural selection and stochastic gradient descent rests on a model which does not include any notion of multiple heredity.

Recovering the equivalence allowing arbitrary population size and recombination

Interestingly, the 'complicating' factor of multiple heredity provides a way to retrieve the equivalence in the presence of multiple contemporary mutations, as long as we continue to consider the limit of infinitessimal mutations.

For a single-heredity population, with multiple contemporary mutant subpopulations, we must either model 'only one winner', or model an ongoing mixture of subpopulations of varying sizes, either of which Ps is unable to model without modification.

On the other hand, in a multiple-heridity population, assuming eventually-universal mixing, and crucially continuing to assume a fixed fitness function (independent of the population mixture), a particular mutation must either fix or go extinct[4].

Proof sketch

So let us consider (instead of xt and xt+1) xt0 and xt1, representing the fixed part of the genotype at times t0 and t_0">t1>t0 respectively, that is the initial genome x0 plus all so-far-fixed mutations.

In the time between t0 and t1 the population will experience some integer number of mutation events (perhaps roughly Poisson-distributed but this is inessential for the proof), each of which is distributed according to Pm. Furthermore, at time t0 some mutations from earlier times may be 'in flight' and not yet fixed or extinct.

Assuming fixed fitness, and infinitessimal mutations, we can represent the probability of fixation by time t1, namely Pf with exactly the same properties as formerly assumed for Ps[5]. Thus each mutation fixed between t0 and t1 satisfies exactly the same unbiased-gradient-sampling property derived earlier, and so, therefore, does their sum xt1−xt0.

This relies on all in-flight mutations not affecting the fitness differential, and thus Pf, of their contemporaries, which is certainly the case in the limit of infinitessimal mutations, but not the case for real natural selection.

Summary of additional assumptions
  1. eventually-universal mixing

In particular, this means no speciation.

NB we also rely on 4. the limit to infinitessimal mutations, in an additional capacity. We also exclude all 'self-play-like' interactions arising from the larger population by relying further on 1. fixed 'fitness function'.

It may be feasible to retrieve a similar equivalence without excluding population-dependent fitness interactions with a different framing, for example considering gradients over 'mixed strategies' implied by population distributions.

Conclusion

Natural selection, under certain conditions, carries out an implementation of stochastic gradient descent. As such, analogies drawn from one to the other are not baseless; we should, however, examine the necessary assumptions and be mindful of the impact of departures from those assumptions.

In particular, two sets of assumptions are presented here which together are sufficient to retrieve an equivalence:

  1. Fixed 'fitness function' or objective function mapping genome to continuous 'fitness score'
  2. continuous fixed-dimensional genome
  3. radially symmetric mutation probability density
  4. limit case to infinitessimal mutation
  5. a degenerate population of 1 or 2
  6. no recombination or horizontal transfer

or, keeping assumptions 1 to 4 and relaxing assumptions 5 and 6

  1. eventually-universal mixing

This is not enough to cover all instances of real natural selection, but provides an approximate mapping from many instances.

Assumptions 2 and 3 together yield 'unbiased' SGD, and in their absence varying degrees of bias arise.

Assumption 1 rules out, most importantly, 'self play' and 'population-based' aspects of natural selection, which have other analogies in machine learning but which are firmly absent from vanilla SGD.

Further work could uncover other parameters of the emergent SGD, such as the variance of the implied gradient, the size of the implicit learning rate, the bias caused by relaxing assumption 3, or quantify the coupling between those factors.

Further scrutiny, especially of the assumptions related to population, 1, 5, 6, and 7, could better quantify the effect of making weaker or different assumptions.

  1. This can be justified in a few ways

    • If fitness is something like an Elo rating then a Boltzmann distribution is implied
    • If we want to extend the two-individual case to the n-individual case but remain invariant to the arbitrary choice of 'baseline' fitness score, then a normalised ratio of exponentials is implied
    • We may further appeal to the maximum entropy property of Boltzmann distributions as a natural choice
    ↩︎
  2. The directional derivative in question is, for ϵ=|ϵ|uϵ,

    lim|ϵ|→0f(xt+ϵ)−f(xt)=|ϵ|lim|ϵ|→0f(xt+|ϵ|uϵ)−f(xt)|ϵ|=|ϵ|(uϵ⋅∇f(xt))=ϵ⋅∇f(xt) ↩︎

  3. Cautious readers may note that the integral as presented is not posed in the right coordinate system for its integrand.

    By a coordinate transformation from Euclidean to hyperspherical coordinates, centred on 0, with ∇f(xt) providing the principal axis, r the radial length, θ the principal angular coordinate, and ϕ the other n−2 angular coordinates with axes chosen arbitrarily orthogonally,

    E[θ|xt]≃∫αsg(r)Ps(r|∇f(xt)|cosθ)θdϵ=∫αsg(r)Ps(r|∇f(xt)|cosθ)θ∣∣∣∂(r,θ,ϕ)∂(ϵ)∣∣∣drdθdϕ=∫∞0∫π−παsg(r)Ps(r|∇f(xt)|cosθ)θ[∫[0,π]n−2∣∣∣∂(r,θ,ϕ)∂(ϵ)∣∣∣dϕ]dθdr=∫∞0∫π−παsg(r)Ps(r|∇f(xt)|cosθ)θ[∫[0,π]n−2rn−1h(ϕ)dϕ]dθdr=∫∞0∫π−παsg(r)Ps(r|∇f(xt)|cosθ)θkrn−1dθdr=∫∞0αskg(r)rn−1[∫π−πPs(r|∇f(xt)|cosθ)θdθ]dr=0

    where we use the fact that the hyperspherical Jacobian ∂(r,θ,ϕ)∂(ϵ) is independent of its principal angular coordinate θ and denote by krn−1 the result of integrating out the Jacobian over the other angular coordinates, and again noting that the symmetrical integral over an odd function is zero. ↩︎

  4. If we do not have a fixed fitness function, and in particular, if it is allowed to vary dependent on the distribution of the population, there are many evolutionarily stable equilibria which can arise where some trait is stably never fixed nor extinguished, but rather persists indefinitely in some proportion of the population. (A classic example is sex ratios.) ↩︎

  5. We can be more precise if we have Pf:R+→R→[0,1] where the additional first parameter represents time-elapsed, so that Pf(δt,δf) is the probability of a mutation with fitness delta δf being fixed after elapsed time δt.

    Here we impose on Pf(δt) (for fixed δt time-elapsed) the same monotonicity requirement over fitness differential as imposed on Ps before.

    The various 'in-flight' and intervening mutations in the proof also therefore implicitly carry with them tm, the time they emerged, and the additional argument to Pf is thus δt=t1−tm.

    In practice we should expect Pf to vary time-wise as a monotonically nondecreasing asymptote, but this property is not required for the proof. ↩︎



Discuss

A Bird's Eye View of the ML Field [Pragmatic AI Safety #2]

9 мая, 2022 - 20:18
Published on May 9, 2022 5:18 PM GMT

This is the second post in a sequence of posts that describe our models for Pragmatic AI Safety. 

The internal dynamics of the ML field are not immediately obvious to the casual observer. This post will present some important high-level points that are critical to beginning to understand the field, and is meant as background for our later posts.

Driving dynamics of the ML field

How is progress made in ML? While the exact dynamics of progress are not always predictable, we will present three basic properties of ML research that are important to understand.

The importance of defining the problem

A problem well-defined is a problem half solved. 

—John Dewey (apocryphal)

The mere formulation of a problem is often more essential than its solution, which [...] requires creative imagination and marks real advances in science.

Albert Einstein

I have been struck by how important measurement is... This may seem basic, but it is amazing how often it is not done and how hard it is to get right. 

Bill Gates

If you cannot measure it, you cannot improve it.

—Lord Kelvin (paraphrase)

For better or worse, benchmarks shape a field.  

David Patterson, Turing award winner

Progress in AI arises from objective evaluation metrics.

David McAllester

Science requires that we clarify the question and then refine the answer: it is impossible to solve a problem until we know what it is. Empirical ML research, which is the majority of the field, progresses through well-defined metrics for progress towards well-defined goals. Once a goal is defined empirically, is tractable, and is incentivized properly, the ML field is well-equipped to make progress towards it.

A variation on this model is that artists (writers, directors, etc.) come first. They help give ideas, and philosophers add more logical constraints to those ideas to come up with goals or questions, and finally scientists can help make iterative progress towards those goals. To give an example: golems, animate beings created from clay, were a common symbol in Jewish folklore, and at times could create evil. There are many other historical stories of automatons creating problems for humans (Pandora, Frankenstein, etc.). More recent stories, like Terminator, made the ideas more concrete, even as they included fantasy elements not grounded in reality. More recently, Bostrom (2002) recognized the possibility for existential risk from AI, and grounded it in the field of artificial intelligence. Since then, others have worked on concretizing and solving technical problems associated with this risk.

For completeness, it’s worth mentioning that sometimes through tinkering people find solutions to questions people were not posing, though many of those solutions aren’t solutions for interesting questions.

Metrics

As David McAllester writes, machine learning and deep learning is fundamentally driven by metrics. There are many reasons for this. First, having a concrete metric for a problem is a sign that the problem has been compressed into something simpler and more manageable (see the discussion of microcosms below), which makes it more likely that progress can be made on it. By distilling a problem into a few main components, it is also far clearer when progress has been made, even if that progress is relatively small.

Unlike human subjective evaluation, most metrics are objective: even if they do not perfectly track the properties of a system that we care about, it is obvious when somebody has performed well or poorly on an evaluation. Metrics can also be used across methods, which makes different approaches directly comparable rather than relying on many different measuring sticks. High-quality datasets and benchmarks concretize research goals, make them more tractable, and can spur large community research efforts. Good metrics also can allow us to detect minor improvements, which enables iterative progress and accumulated improvements.

Metrics that rely too heavily on human evaluation are suspect. First, human evaluation is extremely expensive compared to automatic evaluation, and often requires IRB approval in the case of academic labs. This significantly reduces its utility. Second, human evaluation is slow, which makes feedback loops sparser and thus makes problems far more difficult to iterate on. Third, human feedback is often noisier and more subjective than many automatic evaluation metrics.

For these reasons, the first step working towards working on a new ML problem is to define a good metric.

Limits of theory for machine learning

There can be a tendency for new fields to try to formulate new problems as mathematics problems, but this is not always possible. Unfortunately, if machine learning just isn’t productively formulated as an applied mathematics problem, throwing lots of resources at it as an applied mathematics problem isn’t going to work. Currently, deep learning progress is not well-formulated as an applied mathematics problem.

Consider progress in computer vision:

1960s: Beginnings in artificial intelligence, image processing and pattern recognition

1970s: Foundational work on image formation: Horn, Koenderink, Longuet-Higgins …

1980s: Vision as applied mathematics: geometry, multi-scale analysis, probabilistic modeling, control theory, optimization

1990s: Geometric analysis largely completed, vision meets graphics, statistical learning approaches resurface

2000s: Significant advances in visual recognition

2010s: Progress continues, aided by the availability of large amounts of visual data and massive computing power. Deep learning has become pre-eminent

While some researchers tried to treat image recognition as an applied math problem, as mathematical solutions have many desirable properties, this did not work and they were forced to shift paradigms. Research communities that need to solve problems don’t get to choose their favorite paradigm. Even if the current paradigm is flawed and a new paradigm is needed, this does not mean that their favorite paradigm will become that new paradigm. They cannot ignore or bargain with the paradigm that will actually work; they must align with it. We may want a machine learning problem to be a math problem, but that does not mean it is.

Overtheorizing is a common failure case for solving technical problems, and it happened in AI. Imagine you want to build a mail delivery robot. If you start working on differential geometry, you’re doing it wrong and overtheorizing. It simply is not necessary to solve that problem in order to build the mail delivery robot. The slide with the figure above is from 1995, and argues that many of the subtasks thought relevant for object detection were unnecessary for the task: today, this is obvious. Despite a large number of extremely smart people working on CV as an applied mathematics problem for many years, they did not solve CV, because CV is not an applied mathematics problem.

A similar dynamic happened with what is now known as “good old fashioned AI” using explicit high-level symbolic and search algorithms. While it produced some successes, these inclinations  have been mainly replaced by deep learning in the state of the art.

In natural science, the objective is to discover, study, and explain natural phenomena. Consider optics (1650), thermodynamics (1824), aerodynamics (1757), information theory (1948), and computer science (1950). In natural science, the gold standard is reproducible experiments in controlled conditions. The use of mathematics and statistics is common, since they help provide a theoretical underpinning. In natural science, control over the environment in an experiment is crucial.

In engineering science, the basic idea is to invent new artifacts. Consider the telescope (1608), steam engine (1695), sail boat (BCE), teletype (1906), computer (1941), etc. In engineering, one must use intuition, creative inspiration, tinkering, and trying many things to see what sticks. Invention is often created by accident, by people who created many failing inventions. Some theory is surely needed (a sailboat is far less likely if one does not know that wind can push objects) but it often does not need to be very detailed. Engineering is often bottom-up and creative, and theory does not dictate an exact or even approximate design. For a discussion of deep learning and limits of top-down design, see here. In many cases, natural science follows engineering science, where natural science attempts to explain an artifact created by engineering.

Yann LeCun discusses this distinction further here. The distinction may also be analogized to the distinction between rationalism and empiricism.

DL is mostly engineering science. We have few theories, and the theories we do have provide limited guidance. Attempts at generalizing phenomena in DL are very often not robust. For example, previously l1 loss worked noticeably better than l2 loss for pixel regression, but not recently. Every year, new papers are published that purport to explain the generalization ability of neural networks, but the subject has not been settled. There are some exceptions: for instance, the fact that bigger Transformer models consistently perform better. Since these kind of observations are more robust than theories, and since the phenomena are fragile, the theories are even more fragile. Even ML theorist PhD students will readily admit that theory for DL has not (yet) created action-relevant insights about DL.

DL has many factors that increase the probability of surprises or unknown unknowns: complicatedness, fast changes, ambiguity, opacity, and interconnectedness/multiple causes. In view of this, researchers that work for years towards advancing capabilities to get a specific downstream safety research outcome are not being precautious since this field is not that predictable. This level of unpredictability makes armchair/whiteboard research less likely to succeed; much research must be bottom-up and iterative.

DL methods are often not even obvious in hindsight, unlike more theoretical fields where proofs are self-evident to audiences with an appropriate background. Why do residual connections work? Why does fractal data augmentation help? This property is why tinkering and rapid experimentation is so important.

If hindsight doesn’t even work, that means that it is extremely difficult to achieve foresight into the impact of a proposed project. Instead, it is necessary to fall back on heuristics. First, does the project seem palpably unrealistic on its face? This could be grounds for dismissing it. Otherwise, if there are not yet results, one should instead defer to the researcher’s prior track record, with perhaps some benefit of the doubt given to younger researchers who have not had time to accrue track records.

Creative Destruction in ML

If a temple is to be erected a temple must be destroyed: that is the law – let anyone who can show me a case in which it is not fulfilled!

Friedrich Nietzsche

Like many other fields, ML experiences creative destruction: periods when new and better technology rapidly replaces older methods. This can be related to models for exponential growth as aggregated sigmoid jumps rather than a truly smooth acceleration.

Research, especially research into algorithms, architectures, training techniques, and the like, is often entirely wiped away by “tsunamis.” For example, natural language processing techniques from before BERT are almost entirely irrelevant. A large number of computer vision techniques devised prior to AlexNet have almost no influence on the field today. Lastly, speech recognition became an essentially solved problem with the advent of deep learning. After a tsunami, methods often work out-of-the-box, require less effort to use, and performance is much higher.

Imagine you’re in the late 2000s and care about AI safety. It is very difficult to imagine that you could have developed any techniques or algorithms which would transfer to the present day. However, it might have been possible to develop datasets that would be used far into the future or amass safety researchers which could enable more safety research in the future. For instance, if more people had been focused on safety in 2009, we would likely have many more professors working on it in 2022, which would allow more students to be recruited to work on safety. In general, research ecosystems, safety culture, and datasets survive tsunamis.

By some indications, reinforcement learning is poised for a tsunami. RL currently is extremely unwieldy, requiring careful tuning of a large number of parameters and methods to achieve any reasonable results at all. It often (not always) uses Markovian assumptions and exponential decay, which tends to be replaced by paradigms that better model long-range dependencies (e.g., consider hidden markov models in NLP which have been superseded). Unlike language and vision, RL has not yet been revolutionized by large-scale models. As such, RL appears to have the properties of a field prior to a tsunami. If this is true, it does not bode well for RL safety research methods, which could be washed away.

As a result, developing safety proposals for DL is likely to be a safer bet than RL in both the short and long term. There appears to be a reasonable chance that DL will not be washed away. In this case, it is extremely important to have a high number of people working on empirical methods. If DL is not the last tsunami, research in DL will still aid dataset creation, research ecosystem building, and safety culture for later.

Thinking of methods that will work in the current paradigm and not using this research to help ecosystem building in the larger ML community is doubly bad: it stands the risk of being wiped away by a tsunami, and it didn’t even help research ecosystem building. If one expects more tsunamis, pay more attention to prestige and resources.

The Bitter Lesson argues that there will be more creative destruction and that human ingenuity will matter less and less. Although we do not believe the following scenario is likely, in the long run, AI risk reduction may even be a matter of banal factors: compute, data, and engineering resources allocated towards safety goals, in comparison with other capabilities goals. The amount allocated towards these goals would depend on how important safety is to the system designers, which means safety buy-in among researchers and tech leaders would be a high priority.

The ML research ecosystem

If we want to have any hope of influencing the ML community broadly, we need to understand how it works (and sometimes doesn’t work) at a high level.

Where is ML research published?

The machine learning field, both in industry and academia, is dominated by conferences. Except for a few splashy and publicized industry papers published in Nature, the most important ML papers are all published in a relatively small number of ML conferences. Though there are a few journals, they are not very impactful, nor are papers that are exclusively published at workshops. The vast majority of papers are submitted to conferences.

Prior to publication, most ML research papers are posted as preprints on arxiv.org (pronounced “archive”). Because of the speed that the ML research field advances, it is not sufficient for ML researchers to simply read papers that have been published in conferences, since publication typically doesn’t happen for several months until after a paper is posted on arXiv. Instead, ML researchers need to keep updated on the latest preprints. Many do so via relying on word of mouth or Twitter to indicate which papers are important, while others make more of an effort to identify important papers on arXiv themselves.

Composition of ML Subfields

The size of different machine learning subfields might not be immediately obvious to somebody outside of the field. The graphs and statistics that follow are based on an analysis we conducted of publicly-available data from Semantic Scholar (data is approximate, and is only as accurate as the Semantic Scholar data), and use citations as a rough metric for the size of different subfields.

By far the largest subfield within machine learning is computer vision (CV), with the Conference on Computer Vision and Pattern Recognition (CVPR) being by far the most influential conference in machine learning in terms of the number of citations. Natural language processing (NLP) is much smaller than computer vision, with the largest conference, the Proceedings of the Association for Computational Linguistics (ACL), getting about a fifth the total citations of CVPR. Even smaller than NLP are reinforcement learning and robotics. The most influential conference in this area is the IEEE International Conference on Robotics and Automation (ICRA), which receives slightly more than half the citations of ACL (so an order of magnitude less than CVPR). There are also three conferences that publish many kinds of ML research: the International Conference on Learning Representations (ICLR), Neural Information Processing Systems (NeurIPS), and the International Conference on Machine Learning (ICML). These conferences can contain NLP, CV, and RL, and all rank in the top four conferences along with CVPR.

The top conferences do not gain their influence simply by having many papers. ICRA and IROS publish as many papers as CVPR (Figure 1), but mean and median citation counts in CV and ML conferences (particularly ICLR) are far above those in NLP and RL/robotics (see Figure 3 and 4).

NLP has been growing recently, especially since BERT reduced barriers to entry. For instance, ACL papers from 2020 (see Figure 5) only got 2-3x fewer citations compared with CVPR papers, showing some relative growth of the field (note NAACL and ICCV were not held in 2020). Still, CV was the largest subfield, and RL/robotics has not gained any ground at all.

Explanation of ML Subfields

Microcosms

“Microcosms” in this context are simpler subproblems of harder problems that mirror the larger problems but are more tractable. To give a concrete example, to make an aerodynamic bicycle for a race, you might first want to start by making it aerodynamic in a wind tunnel. Although the conditions of a wind tunnel are simplified compared with the real world, they are similar enough to yield useful insights. Some simpler problems are not that microcosmic, however, because they may make too many simplifying assumptions and thus not be representative of the macrocosm (e.g., some gridworlds).

In general, the majority of research inquiry is conducted on microcosms. Work on these problems can inform us about the future or even directly influence future systems, as some current ML algorithms are highly scalable and may be a part of long-term AI systems. Next, we will discuss two of the most important microcosms in machine learning.

Why do DL researchers look at ImageNet and image classification so much?

Historically, CV and pattern recognition in general have gone together. The ImageNet dataset, an old compilation of images and associated labels for those images, continues to drive the field. Why? It is a good microcosm. Researchers have found that performance on ImageNet is highly predictive of downstream performance in numerous applications like segmentation, clustering, object detection, and downstream image recognition. Many researchers also view image understanding as a problem that is upstream of video understanding, which is important for a whole range of additional applications.

Deep learning researchers use CV as a representation learning proxy, not because they are particularly interested in classifying cats. Most DL building blocks (some activation functions, batch normalization, some optimizers, dropout, convolutions, residual connections, etc.) have emerged from researching image classification, so it is a useful whetstone to sharpen DL algorithms against. Consequently, people do not think they’re just researching vision but instead researching how to do representation learning and deep learning in general. There are, of course, exceptions: semantic segmentation, depth maps, and downstream applications of CV do not necessarily help with general representation learning.

Many economic incentives and funding sources are available for CV because vision is useful in many industries, which keeps the field large.

CV is also large because it’s very possible to improve image classification performance with ideas since scaling is less important. Many methods do not currently have consistent returns to scale: for instance, current masked autoencoders do not consistently perform better with more data or compute.  Most researchers, especially academics, are heavily incentivized towards fields where improvements can be made with ideas, because ideas can be found by competent researchers with good taste who spend enough time tinkering. Meanwhile, in NLP, a larger share of progress is made by simple scaling, which makes for less interesting work for researchers, and less incentivized outside very large labs with access to compute resources.

In DL, findings for some data transfer to other types of data. This is partly because different kinds of natural data have similar underlying statistical properties. Consequently, studying how to do representation for some data distributions often transfers to other data distributions. Many techniques, such as residual connections, that helped with ImageNet helped with discrete signals (text) and other continuous signals (speech).

Why is NLP the second largest field?

As detailed above, image classification is not about images as much as it is about general capabilities to analyze continuous structured signals. Likewise, natural language processing is important because it studies discrete signals.

In recent years, NLP and CV have started to coalesce, with more multimodal models being able to process both continuous and discrete signals. In addition, insights from NLP, such as large pre-trained transformer models, are now percolating into CV (e.g. vision Transformers), as techniques have become more general and differences between the two paradigms are decreasing.

Issues with Conferences

The conference review process has serious flaws. For instance, best paper awards mean very little; oral or spotlight designations are not highly predictive of longer-term impact either. In ML conferences, awards and designations are highly biased towards theory papers. In vision, these awards have a strong political element, where some awards are sometimes given to researchers that are seen as needing or deserving of a career boost.

In 2014 at NIPS (now called NeurIPS), an experiment was conducted where the review committee was split in half, and 10% of papers were assigned to be reviewed independently by both committees. 57% of papers accepted by one committee were rejected by the other committee, and vice versa. In comparison, given the overall acceptance rate, the rate would be 77.5% if decisions were purely random and 0% if decisions were perfectly correlated.

A recent analysis found the correlation between reviewer scores and eventual citation rate was very weak at ICLR 2020, after controlling for acceptance decisions. The most transformative ideas are often initially disputed or doubted; to be accepted, consensus often needs to be reached in a short discussion period. Additionally, reviewers adversely select for complicated methods (neomania) and disincentivize simple methods that work, even if impact is one of the dimensions they are supposed to be evaluating. In ML conferences, they incentivize theory, which these days usually is anticorrelated with ultimate impact.

As pointed out by Yann LeCun (timestamp 2:15), a large number of young reviewers just want to point out errors rather than assess the paper overall. However, correcting errors is unlikely to make a paper go from low to high impact or help researchers write better papers the next time around.

These experiments cast doubts on the review process, even though researchers continue to act as though conference selection processes are wise. This doesn’t mean that the review process is not at all useful. For one, conferences provide comments to papers, not just scores. More importantly, anonymous peer review is necessary for researchers in their first five years of research, as they need tough comments from random community members; without peer review, people will rarely hear what people truly think since specific disparagement is highly uncommon in other CS contexts. Reviewer comments also limit parochialism, an increasingly large problem as the field becomes more expansive. Reviews also require papers to have some level of technical execution ability; if they’re below a threshold, most people do not even submit the paper. Lastly, it’s important to consider the effect that the anticipation of the review process has on researchers. Even if reviews are a noisy process, researchers will be frequently thinking about how to make sure their paper is accepted by reviewers, and in many cases this can encourage them to have a stronger paper.  Like democracy, the review process is not a perfect system, but it works better than alternatives (e.g. “trial by upvote”).

One implication of the flaws with the review process is that new approaches cannot reliably be evaluated just by thinking about them. Rather than rely on experts to evaluate new works, communities need the test of time, the ultimate filter for bad ideas.

Consequential and Inconsequential Papers

Marked progress on a specific problem, even if you have a field working on it, is usually fairly infrequent, where there will be a paper that actually moves things ahead every 6 months to 3 years. In the meantime, there are many unpublished and published papers that are inconsequential. It is very hard to come up with something that actually helps.

How are so many inconsequential papers published? They may have successfully presented themselves as consequential. For instance, it is common for researchers to publish papers in which they achieve state of the art performance in one metric (e.g. out-of-distribution robustness) while sacrificing performance in another metric (e.g. calibration) compared to the previous state of the art. This might be accepted, because it appears to make progress in one dimension. The best papers, on the other hand, make Pareto improvements, or at the very least make extremely lopsided improvements where a minor sacrifice in one dimension leads to a major gain in another.

Another way that inconsequential papers might get published is for authors not to publish results on all of the relevant datasets. For instance, sometimes people publish results of evaluating on CIFAR-10, but not CIFAR-100, often because they are not able to satisfactorily perform on the latter. This sometimes goes unnoticed by reviewers.

Finally, papers that might seem consequential at first turn out to be inconsequential when they are wiped away by tsunamis.

Interestingness

In many cases, ML researchers are motivated by interestingness more than usefulness. This is related to the fact that those who choose to enter research have high openness to new ideas and frequently seek out novelty. This can bias them against research that appears more “boring,” even if it has much more practical relevance.

The bias towards interestingness becomes even more extreme in areas with less concrete metrics for success. In such cases, researchers can sometimes get away with publishing research that is empirically not very useful, but is interesting to reviewers. This is the reason behind many ML “fads,” which can last many years. 

Historical Progress

In order to begin to attempt to extrapolate into the future, it’s necessary to understand historical progress. In this section, we will provide some examples of historical progress, using metrics defined for the field. For many metrics, especially those that are highly researched, progress is roughly linear (or log-linear). For less-studied problems, there can be sudden jumps.

Data for the graphs below is from Papers With Code, with some charts consisting of adapted data (mostly to present log error rates rather than accuracies).

Image Classification

ImageNet has been one of the most influential datasets, and image classification has remained the most important benchmark in computer vision. Frequently the top-1 or top-5 accuracy on ImageNet is reported, but it’s also useful to look at the log error rate since progress has been roughly linear on that metric. 

Video understanding

The following chart shows progress on action recognition in videos.  The state of the art in 2018 was the temporal relational network, which was a variant of a convolutional neural network. In 2022, the state of the art is a vision transformer that was pretrained on a separate dataset. Progress has been relatively slow in video understanding despite strides in image understanding. One heuristic is that video understanding is ten years behind image understanding.
 

Object Detection

Object detection is sometimes thought of as dramatically accelerated by deep learning. In reality, while there was a period of 1-2 years in which progress seemed to be stalling prior to deep learning, deep learning merely continued the existing trend in capabilities improvements [1,2]. Deep learning did revolutionize the methods in object detection, and the old object detection paradigm was washed away. Starting in 2017, however, performance once again stalled, leading to a plateau that has been longer than any pre-deep learning! In fact, there was nearly no progress until 2021, when a paper made a relatively large improvement. The paper that did so leveraged pre-training with data augmentations.

Image Segmentation

Image segmentation (the labeling of different pixels of an image) on the COCO progressed extremely quickly between 2015-2016, but it has leveled off.

Adversarial Robustness

Progress on adversarial robustness has been fairly slow. See the following graph:

Language Modeling

Perplexity measures the ability of language models to be able to predict the next word given a sequence of words (a lower perplexity is better). Perplexity on the Penn Treebank dataset has been decreasing over time in a roughly linear way.

Translation

Below is one example, on WMT2014 English-German.

Question Answering

Question answering on the SQuAD 1.1 dataset progressed quickly until 2019, when it leveled off.

Emergent Properties

The view that machines cannot give rise to surprises is due, I believe, to a fallacy to which philosophers and mathematicians are particularly subject. This is the assumption that as soon as a fact is presented to a mind all consequences of that fact spring into the mind simultaneously with it. It is a very useful assumption under many circumstances, but one too easily forgets that it is false. 

Alan Turing

The qualitative impact of an order of magnitude increase in parameters, or a new algorithm, is often difficult to predict. Capabilities can sometimes emerge suddenly and without warning.

For instance, the effect of additional parameters on image generation is not immediately predictable:

Scan the images left to right, blocking the rightward images. Try and predict how good the image will be after an order of magnitude increase.

BERT could not be easily tuned to do addition, but RoBERTa, which was just pretrained on 10x more data, can.

AlphaZero experienced a phase transition where internal representations changed dramatically and capabilities altered significantly at about ~32,000 steps, when the system learned concepts like “king safety, threats, and mobility” suddenly. This can be seen by looking at the system’s preferred opening moves, the distribution of which dramatically changes at 32,000 steps.

One paper showed that in some cases, performance can improve dramatically on test data even after it had already saturated on the training data:

In a paper studying the Gopher model, gold labels initially did not improve performance on the FEVER fact-checking dataset by much at smaller model sizes, but had significant impact at larger model sizes.

Anthropic’s Predictability and Surprise paper demonstrated fast emergence of addition, multitask understanding, and program synthesis as the number of parameters increased.

It will be hard to make systems safe if we do not know what they are capable of. Additionally, it will be hard to foresee how systems will behave without empirically measuring them.

Lastly, rapid changes can be caused by creative destruction. Most of these graphs did not even start prior to a decade ago, because deep learning ushered in an entirely new paradigm for solving many of the problems above.

Notes on historical progress

In vision, state of the art algorithms usually change year-to-year, and progress has been driven by algorithms and compute. In contrast to this, in NLP, once we had Transformers, the underlying architecture seemed to stabilize. In NLP, it’s currently difficult to have something work better across a broad range of classification tasks without just scaling data or compute.

Compute has been growing rapidly since the era of deep learning, although the rate of compute increase for very large scale models appears to be lower. See the graph below, from this paper.
 

The scaling laws paper showed how compute scaling can drive performance in language modeling, and how this relates to optimal dataset size and length of training. The Chinchilla paper was a good example of using scaling laws to make decisions that resulted in better performance; it achieved superior performance to models trained with a similar amount of compute by reducing the number of parameters and increasing the amount of data.

Analyzing the trajectory of researchers

We have just provided some background on the trajectory of the overall ML field. In this section we will discuss how to analyze the individual trajectories of ML researchers.

Bibliometrics

Citations are not a perfect metric for an individual’s influence on the ML field, and sometimes it is claimed that they are not useful as a metric at all. Of course, not everything of value is captured in citations: whether somebody gives impactful talks, is a good mentor, and so on is not reflected in citations. At the moment, however, there is no alternative set of metrics which better capture true influence and can be scaled up to thousands of people who do not know each other. The ML field at large uses citations as their metric of choice, and this has not prevented the field from being highly competent at advancing capabilities.

h-index (defined as the largest n where the author has at least n papers with n citations)  benefits academics who publish many papers, which is not necessarily a good indication of their true influence. A better measure is Semantic Scholar’s Highly Influential Citations, which measures citations that are not cursory (Was the paper cited multiple times? Was the paper cited as “using the method from Doe et al.” instead of just included as background?). Many top industry and top academic researchers are similar by this metric, even though academic researchers output a far higher volume of papers. Top industry researchers publish fewer papers, as industry researchers divide their research into papers differently from academics, but the best industry researchers have a total impact that is not dissimilar from the best academics.

Note that bibliometrics is a metric for influence on the empirical machine learning community. Influence is not perfectly correlated with value: not all papers that influenced the community are that valuable to the community or even those that cited them. However, influence is a good proxy for how much a researcher understands what is used by the community.

Some industry people have many citations, but it’s because they were on a big lab paper. For example, a researcher can do well metric-wise if they are on the tensorflow paper, even though there are several dozen authors on the paper.

Research ability and impact is long tailed

The most impactful researchers in ML, as in other fields, make up the vast majority of the impact in the field overall. Most researchers, even those who are smart and able to publish, have very little impact on the field. For instance, when measuring impact by the number of highly influential citations, about ~4.5 orders of magnitude separate typical PhD students who have published papers so far (~1) and the topmost ML researcher (Kaiming He, who did ResNet and Mask R-CNN, with ~30k). Of course, about 1-2% of applicants to Berkeley’s AI PhD program get in, so we’re describing a large gap between typical middle-stage Berkeley PhD students and top researchers—the gap between top researchers and other researchers without any influential citations is even larger. Even among PhD students at top universities, there is an extremely wide spread: some may graduate with 1,000 highly influential citations while others graduate with just a handful. While there are some 10x engineers, there are ~1,000x researchers (this is an even higher multiple when looking at total citations).

Figure 6 shows citations for papers published at the 12 conferences mentioned in the previous sections. We do not have accessible data on highly-influential citations, so we have to rely on all citations. The distribution of citations for papers is long-tailed. For papers published in 2016, the top 1% of papers accumulated 52.7% of the citations, the top 0.1% of papers accumulated 30.4% of the citations, and the top paper (out of 5155; the Adam paper) received 11.6% of the citations. The bottom 50% of papers accumulated only 2.8% of citations. This is expected for long-tailed distributions. Figure 7 shows the same statistics for CVPR: the skew is not simply due to inter-conference variation.

For individual researchers, impact is also extremely skewed (see Figures 8-11). Looking at researchers with at least one first author paper published in a top conference between 2016 and 2020, the top 1% of researchers accumulated 35.5% of citations on first author papers (papers they led research on) in those years, the top 0.1% accumulated 13.6% of citations, and the top researcher (out of 26,381; Kaiming He) accumulated 3.5% of citations. Although there is a skew in the number of papers published, it is comparatively small (less than two orders of magnitude) and it cannot account for the six orders of magnitude between the top researcher and the 1,243 researchers receiving only one citation. Researchers don’t get far more citations simply by publishing more papers.

Some of this, of course, is researchers getting lucky and managing to publish a great paper. However, not all success should be dismissed this way. The plot below compares log citations researchers received for first author papers published in top conferences between 2016 and 2017 and papers they published between 2018 and 2020. It only includes researchers who published their first first-author paper in one of these conferences in 2015, so as to control for the effect of seniority. There is a noticeable correlation of 0.55.

Given the vast differences in ability, it makes sense to pay most attention to those who have demonstrated an ability to have extreme impact, since they are likely to capture the vast majority of the impact.

Given this context, it is extremely important to be able to differentiate researchers based on their impact and ability level. For long tail distributions, the impact of the top few papers are able to approximate the sum of the impact of all the papers. Practically, most researchers can be considered to have essentially no impact. Next, we will discuss heuristics for determining which researchers do have impact.

Transfer between paradigms is hard

Creative destruction in ML makes many top researchers no longer effective. Often professors were selected based on skills that used to be needed, but are no longer useful for cutting-edge research. This means that professors with many citations or professors who once made important contributions to the field may no longer be relevant.

For example, some universities are currently lagging behind competitors because they hired a high proportion of traditional ML researchers. However, the skills useful for researching SVMs, Bayesian nonparametrics, and probabilistic graphical models are math, optimization, and statistics. These are not very useful in the context of deep learning. Improving deep learning algorithms comes from messy bottom-up atheoretical empirical tinkering, not precise top-down armchair/whiteboard rational contemplation. The best DL researchers hone their tastes, use their gut, and run many experiments and see what sticks, as it’s very difficult to tell a priori which ideas will improve performance. It’s nearly always faster to run the experiment rather than think about what the result will be, in part because the base rate of methods that actually help is extremely low. Some DL researchers say “good ideas are a dime a dozen” because researchers have to try out many reasonable ideas before one sticks. This doesn’t mean that ideas are unimportant, but it does mean that they are not sufficient. In DL algorithm improvements, a lot of research is governed by underspecified, vague intuitions, while math people tend to resist that and thus have difficulty transferring to deep learning.

Most ML researchers have switched to DL, but most of the top forces of the past are no longer standout forces because their skills have not transferred. However, there is an exception: some older computer vision professors (e.g., Jitendra Malik, Trevor Darrell) are still standout researchers advancing the state-of-the-art; they improved pattern recognition techniques for images and developed skill sets similar to what is needed for deep learning.

The lesson is that extreme accomplishment is quite fragile (see the tails come apart). For example, Michael Jordan (the basketball phenom) wasn’t able to become a top-tier baseball player. Different niches require different skills. Accomplishment is not merely based on general research ability or on intelligence, but also significantly on specialized research ability. An implication is that you can’t take researchers who are highly impactful in one area and by default expect them to be impactful in another area. Given that most ML researchers didn’t transfer well to DL, it is difficult to expect those from further-removed fields to transfer.

This is not to say some people do not have generally transferable abilities. For example, Yann LeCun had a large impact in deep learning, and he also created a notable file compression format. You may see somebody who has been working in a subfield that has suddenly blown up, but was previously not thought to be important. Were they prescient, or just lucky? To answer this, again look for indications of generality. Have they been right in a big way more than once? Have they had an extreme impact in two or more distinct areas? If they’ve demonstrated ability in multiple areas, then that can serve as evidence for transfer ability. As this is so rare, often the best we can do is select people who have demonstrated extreme competence in their specific research area. Next, we will discuss some of the underlying dynamics that may distort researchers’ track records or the quality of their advice.

The Matthew Effect

To those who have, more will be given. 

—Matthew 25:29

The Matthew Effect, so named after the biblical verse, is an extremely explanatory dynamic. The visible accomplishments of researchers in a field is highly path-dependent on their initial success early in their careers. Researchers who had some success in grad school, perhaps due to sheer luck, will be more likely to be able to work at a top institution. Once they are there, they will be more likely to be able to recruit better grad students, attain more funding, be invited to more talks, be thought smarter than others, and have their papers read more widely, even if their research is no longer useful. Many professors got unusually lucky (on top of some general level of relatively high competence) and their success became inevitable when they became appointed, even if they do not have extremely high research ability; some could have ended up like a 1 sigma above average research scientist. Similarly, a graduate student who has had a successful paper is more likely to get better advisors, fellowships, access to compute, better collaborators, more citations due to their paper’s Google search ranking, and so on. Success is disproportionately rewarded, and this must be recognized when evaluating researchers or papers.

Suppose a researcher wrote a fundamentally flawed paper that created a temporary “fad” in machine learning. For a couple of years, they managed to gain many collaborators and attach themselves to a large number of well-cited papers even if they were not the intellectual driving force of those papers. Later, they were able to get a management position in an industry lab owing to their status, where they leveraged compute resources and headcount to publish more papers without having much of an intellectual influence on their team. Such a researcher would be highly eminent, even after the initial fad was over, due to their position. However, after correcting for the Matthew Effect, we shouldn’t expect them to have exceptionally high research ability. To fully evaluate research ability, one must divide by the resources available to that researcher, which in this case have been skewed.

Incentives

Whenever interpreting somebody’s beliefs, one must ask how their incentives align with that belief. Researchers, like most people, frequently fall prey to motivated reasoning. Here are some examples of incentives and motivated reasoning:

  • Researchers who are good at math are incentivized to enjoy theory and think theory is important, because they are better at it. They are also incentivized to characterize rapid, successive empirical advancement as meaningless “benchmark chasing,” or many ML successes are just “glorified pattern matching, curve fitting, and memorization” not addressing “the fundamental problems.” They’re incentivized to work on what they can understand as opposed to work on what performs well; they’re incentivized to search in the space where their mathematics can help, which is not a comparatively large space (this is analogized to the streetlight effect).
  • Researchers with natural science backgrounds are incentivized to believe that DL is more of a natural science problem than an engineering science problem (this distinction is covered below).
  • Researchers who don’t have a large research budget are incentivized to believe that scaling and compute is less important, because they are unable to pursue this line of research themselves, and therefore they are incentivized to believe that timelines are longer.
  • Researchers who have not been able to produce high-quality deep learning research will be more likely to think a paradigm shift needs to happen away from deep learning, because that could be favorable to whatever they’re researching.
  • Researchers in labs that are currently highly impactful will be less likely to think a paradigm shift needs to happen, because this might throw off their success or make their work irrelevant.
  • Researchers working in a very specific area are incentivized to believe that their problem will require many more years of research, because they do not want to consider pivoting. They’ll rarely say that their field is almost solved or that something they’re not working on from another subfield will automatically solve it.
  • Students are incentivized to believe that their advisor’s outdated line of research is important even if there isn’t evidence that there will be any kind of comeback (e.g., convex optimization).
  • EAs and rationalists are incentivized to believe that the most important research is happening within their communities, because that is where most of their connections are.
  • Researchers who primarily work on improving general capabilities are incentivized to believe that we don't need to worry about catastrophic risks from AGI, because the alternative is to feel guilt or (possibly at great personal cost) switch research topics.
Grad Students

In ML, grad students at top places (e.g., Berkeley, Stanford, MIT, CMU, …) publish a median of two lead author papers at top conferences, and they typically stay in grad school for 5-6 years. That means they lead one paper around every three years, suggesting the extreme difficulty of doing publishable research. Most grad students at top universities are not particularly good at research; that’s because outcomes are long tailed, and in creative endeavors few do well. These systems are, for better or worse, not egalitarian. It’s important to note that PhD students are not selected based on how good they are at their job (research). While research output in undergrad is considered in PhD admissions, students can be (co-)first authors on great papers merely by being technically adept and working in a high-impact lab. These skills are not enough for PhD students, who must do idea generation of their own. Most of their selection is based on recommendation letters, with some GPA constraints. Since undergrads do not have much of a track record, selection is very noisy. This is also true for fellowships, which are often awarded before applicants can amass a track record with a clear signal.

A few students publish far more than two lead author papers. These students are substantially more visible, leading people to think nearly all graduate students publish many papers. For example, a typical graduate student who becomes a professor at a top university usually has at least 8 papers at the time of graduation, and it’s common for them to have ~10 papers (the current maximum is 18). (Caveat: this describes papers in ICLR, ICML, and NeurIPS; NLP researchers tend to have a somewhat higher count.)

Students are usually heavily influenced by their advisor, and work on topics that their advisor is working on, which is usually what their advisors have grants for. Despite this, at top places students usually don’t get much advising, with some only meeting with their advisor for an hour once a month. Even if students meet with their advisors infrequently, they tend to emulate them in their head while researching. Additionally, students also often learn much from older graduate students and postdocs under their advisor.

Most funding for academic research is from government agencies. It’s common even at top places for profs to have highly restrictive, high-pressure DARPA grants, which functionally restricts what students work on. Awards from industry are rarer and usually quite small (e.g., 50K or 100K, with some exceptions). Meanwhile the median CS NSF grant is very roughly $500K. For reference, in the US, grad students cost ~100K/year (so on average more than half a million throughout their PhD).

AGI and AI Safety

The purpose of understanding the ML field is so that we can make progress on reducing existential risk, so we will now speak specifically about AGI and AI safety. In this section, we only discuss the current state of these topics within the broader ML field, but in the future we will address how progress can be made.

Nearly all ML research is unrelated to safety

At NeurIPS 2021, about ~2% of all the few thousand papers are safety-related (Dan counted manually). From the 2021 NeurIPS call for papers, below, the safety-related keywords are bolded. The vast majority of keywords are not safety related.

  • General Machine Learning (e.g., classification, unsupervised learning, transfer learning)
  • Deep Learning (e.g., architectures, generative models, optimization for deep networks)
  • Reinforcement Learning (e.g., decision and control, planning, hierarchical RL)
  • Applications (e.g., speech processing, computational biology, computer vision, NLP)
  • Probabilistic Methods (e.g., variational inference, causal inference, Gaussian processes)
  • Optimization (e.g., convex and non-convex optimization)
  • Neuroscience and Cognitive Science (e.g., neural coding, brain-computer interfaces)
  • Theory (e.g., control theory, learning theory, algorithmic game theory)
  • Infrastructure (e.g., datasets, competitions, implementations, libraries)
  • Social Aspects of Machine Learning (e.g., AI safety, fairness, privacy, interpretability)

We probably want >30% of all ML research to be safety-related.

How do researchers feel about AGI and AI Safety? 

This section is based mainly on Dan’s personal experience. There hasn’t been a survey on experts’ views on AI safety for a while, so personal impressions are the best we have to go on.

Researchers are generally quite technopositive and hope that their work will make a huge positive impact in the world. Much of this tendency is borrowed from the tech industry, which is famously utopian. Likewise, many act as though we must progress towards our predestined technological utopia, and it cannot arrive soon enough. These feelings are amplified in AI because it is perceived to be the next major technological revolution, and researchers want to be part of this.

AI winter made it less acceptable to talk about AGI specifically, and people don’t like people talking about capabilities making it closer. Discussions of AGI are not respectable, unlike in physics where talking about weirder long-term things and extrapolating several orders of magnitude is normal. AGI is a bit more like talking about nuclear fusion, which has a long history of overpromises. In industry it has become somewhat more acceptable to mention AGI than in academia: for instance, Sam Altman recently tweeted “AGI is gonna be wild” and Yann LeCun has recently discussed the path to human-level AI.

In general, the aversion to discussing AGI makes discussing risks from AGI a tough sell.

Moreover, since safety researchers have not contributed many quality works, their contribution to complaint ratio is low, which makes the community fairly unlikable.

Safety and value alignment are generally toxic words, currently. Safety is becoming more normalized due to its associations with uncertainty, adversarial robustness, and reliability, which are thought respectable. Discussions of superintelligence are often derided as “not serious”, “not grounded,” or “science fiction.” Read a debate about safety between Stuart Russell and Yann LeCun here.

Ways ML researchers want to advance capabilities

Below we will detail different paths to capability advancements that are commonly promoted in industry and academia. None of these paths are original, and they’ve all been publicly discussed by non-safety AI researchers. 

We need better environments. Humans became intelligent because their environmental conditions incentivized intelligence. The environments currently used to train models frequently do not (for instance, they might instead incentivize memorization, the ability to predict language, or chess-playing ability). In this view, the true bottleneck to intelligence is being able to design diverse environments with the proper incentive structure. DeepMind’s Reward is Enough is an example of this claim, and hypothesizes that all major cognitive abilities, including language and imitation, can arise from “the maximisation of simple rewards in complex environments.” This hypothesis suggests that there will be a surge in the performance of reinforcement learning, which is the most natural paradigm for systems that must interact in environments.

We need a neurosymbolic AI paradigm shift. Some researchers, especially those who are older and have more mathematical inclinations, believe that systems purely based on deep learning will sooner or later hit a wall. They believe that deep learning is merely picking up on correlations, rather than building true causal models, and that explicit structures are needed to attain causal reasoning. We need “quantized, discrete representations.” Since planning in current supervised/self-supervised models is non-explicit and not tested in the real world, you will never be able to ask a model to “build a battery manufacturing plant” and expect it to be successful. This view implies that timelines will be longer, since there will need to be another major paradigm shift before AGI can happen.

Solving math or programming lets us bootstrap intelligence. This rests on the assumption that for certain kinds of tasks, there is an ability level which, after it is reached, allows the agent to rapidly improve itself. Math could be one example of this, because higher-level math can be directly proved from lower-level math just with sufficient logical reasoning skills. In addition, it is relatively easier to verify proofs than it is to generate verifiable proofs, which means that automated checking is possible. Similarly, code generation could also have this property, because complex codebases are built with simple lower-level components. Programming also has more available data than math, and in many cases is even more easily verifiable than math is. The underlying idea is that once the knowledge of math or code generation reaches a certain level, the system can engage in self-play and improve itself.  Christian Szegedy, who leads math ML research at Brain, argues that one plausible path towards bootstrapping is autoformalization, where natural language is converted into verifiable mathematical constructs. David McAllester writes that AlphaZero succeeded because it relied on perfect simulation, which is something we can’t have for most tasks (e.g. language). He claims that because we do have a form of perfect simulation in mathematics, this is most likely to be the dominant paradigm. This approach could also potentially be used to bootstrap superintelligence from AGI, in addition to producing AGI.

Just have a bigger model. This view argues that every important capability will arise from a larger and more general upstream multimodal model. Proponents typically point to what they see as increasingly general and compute-intensive progress in vision, language, and RL. The implication is that ideas are less important than capital, engineering expertise, supply chain, and physical infrastructure management. This is appealing to those without top research ability, and it is less appealing for those who have a track record of generating useful ideas. This view frequently makes reference to scaling laws, which show that simply scaling compute, data, and model size is enough for performance increases for transformer models. Richard Sutton’s Bitter Lesson is a more general articulation of this argument: that learning and search based systems relying on computational power have always outperformed systems with more carefully designed inductive biases. However, it’s important to note that Sutton is not simply referring to scaling up compute, but also developing algorithms that effectively exploit compute.

Note that continued scaling will be difficult.

An additional comment on scaling is that perhaps it can take you to average human-level performance, but being superhuman in a domain does not follow. In both humans and ML models, we could imagine a general factor g that helps learners learn tasks more quickly and do better on them. In humans, IQ is a proxy for g; in models, it can be argued that parameter count is a proxy for g (models of larger size converge faster and better).  g does not determine all cognitive capabilities. Plenty of people have high IQs but, for all that, they do not automatically have the skills needed to be great researchers, great writers, etc. The argument for scaling may be an argument for general cognitive ability, but this isn’t all you need (it explains some of the variation in human outcomes, but clearly not all).

Lastly, specialized architectures can often vastly outperform more general architectures. MuZero will probably continue to crush very large versions of today’s Transformers at Go. AlphaFold uses far less computational power than the largest Transformer models, but can predict protein structures, a task which more general-purpose Transformers are entirely incapable of. Some problems are better suited for code than neural networks, such as calculators.

Better upstream representations are all that matter and everything else will follow. RL is far too sample inefficient to work well currently, because it typically does not rely on any prior knowledge in building its representation of the environment. We can fix this with self-supervised learning, which can learn the distribution rather than an optimal policy, and can do so with far worse-quality data. In addition, it might be possible to learn a distribution for many environments at once, with minimal fine tuning for individual environments, much as upstream Transformer models are finetuned on individual tasks. Under this view, approaches like the decision Transformer will become more common as we attempt to elicit optimal policies from already-learned distributions rather than explicitly learning optimal policies throughout training. It is frequently assumed that better upstream representations will require better scaling. Eric Jang gives arguments for this view, which is currently popular at Meta AI Research and Google Brain. Yann LeCun also gave a talk on the subject recently.

AGI Timelines

Researchers’ Timelines

In 2016, expert forecasts for human-level AI varied extremely widely, with a median estimate of about 50 years. They forecasted a number of narrow AI milestones beforehand, such as translating languages (2024), autonomous vehicles (2027), and working in retail (2031).

At the MATH AI Workshop 2021 (timestamp 5:32:38), some researchers forecasted how long until an AI is at the level of a human-level mathematician. Yoshua Bengio (one of the pioneers of deep learning and Turing Award winner) predicts a human-level mathematician will be achieved in 10 years. Timothy Gowers (a mathematician and Fields Medalist) said going from math olympiad problems to professional mathematics should be a very quick process and said 20 years in total. Christian Szegedy (a DL researcher, math PhD, Batch Norm creator and adversarial examples discoverer) said 8 years.

For a long list of claims made about AGI timelines, see here. In the most recent decade, peoples’ timelines have usually been decreasing.

Next, we will present arguments for both shorter and longer timelines, representing the most common views in either direction. We do not necessarily believe these arguments.

Arguments for longer timelines

There are three major bottlenecks to AGI, which cannot be resolved with the current deep learning paradigm. As a result, AGI will not arrive until there is a paradigm shift.

Robustness

Robustness is a major problem with deep learning systems, and will need to be resolved before they can be practically useful in cases where extreme reliability is necessary.

We cannot solve robustness by simply trying to make everything in-distribution, for several reasons. First, we cannot sample points from the future, so it is always out-of-distribution. Second, adversarial examples are already in-distribution for adversarially trained models, but Transformers are not adversarially robust. Mere scaling trends will not allow us to reach 99.9% accuracy soon, so extreme reliability is not near.

Scaling might not buy robustness to long-tail scenarios. For example, leveraging existing massive datasets is not enough to solve autonomous driving, as models trained with Internet data and petabytes of task-specific driving data still are not robust to long tail road scenarios. Right now, some state-of-the-art learning algorithms do not see much benefit from further scaling.

Sequential Decision Making

Despite great progress in deep learning, it still struggles with sequential decision making, which will be necessary for AGI because the most important applications of AGI will need AGI to replace humans over long time horizons, not merely provide them knowledge or information. Likewise, standard reinforcement learning techniques have also progressed quite slowly. Until reinforcement learning or deep learning can learn to do well in sequential decision making, which doesn’t seem to be close, there will not be AGI.

Generality

If you extrapolate performance on MMLU (a multiple choice test spanning 57 subjects including law, physics, medicine, accounting, etc.), it appears that 100x of compute will be necessary to attain full performance. Using this much compute would lead to a wide range of engineering challenges, and the Common Crawl dataset becomes too small to be used. 

However, even 100x compute is not enough. We will likely need more compute for autocompletion, rather than just multiple choice selection, and it is plausible that scaling laws will slow down, so we can (charitably) add another 10x. The result of this is that we will need 100 million to 1 billion GPUs. At current costs, this is $1-$10 trillion. We are nowhere close to that level of investment.

Lastly, it’s important to note that in the case of longer timelines we might have far greater ability to solve the essential safety challenges, so we may wish to act as though timelines are longer.

Arguments for shorter timelines

Algorithmic improvements are possible

Arguing for AGI through compute scaling alone is difficult, but we are likely to see large algorithmic improvements. For instance, DeBERTa v2 beats T5, and it's an order of magnitude smaller. DeBERTa v3 exceeds the performance of GPT-3 (finetuned) with 2.5 orders of magnitude fewer parameters. Chinchilla is another example of a massive improvement that did not come from scaling compute. On GSM8K (eighth grade arithmetic word problems), chain of thought and self-consistency methods drive performance from 18% to 75%.  The current LM objective (predict the next word/masked words) is extremely simple. Improvements to the objective could potentially yield performance gains at no additional cost.

Bootstrapping could keep progress going

As detailed above, bootstrapping environments like programming and math will allow for rapid improvements driven by the fact that outputs in these areas are easily verifiable. Currently, there does not appear to be danger of an AI winter. Progress in areas like programming, law, medicine, and autonomous vehicles will likely keep investment increasing, which will power additional scaling.

We may be surprised by emergent capabilities

ML has some degree of traction on almost every cognitive task that a person can do, and once the level of an average person has been reached, it will not be long before elite performance is achieved. Emergent capabilities jumps have often appeared at the same time. This pattern may continue, and if it does, we may see sudden changes in many capabilities at once, which could be surprising.

Lastly, it may be better to err on the side of believing in shorter timelines: if we are surprised by capabilities that come faster than expected, this is far worse than if we have surprisingly more time.

Summary

This post has presented a bird’s eye view of many of the most important properties of the machine learning field. To summarize:

  • Metrics are essential for creating progress in the ML field and are the first step for introducing or growing new subfields.
  • The researcher does not choose the paradigm; the problem chooses the paradigm. Math cannot solve a problem that isn’t a math problem. 
  • Computer vision and NLP are the largest subfields within ML because they serve as good microcosms for continuous and discrete signals.
  • Research impact is power-law distributed. Even though all researchers are very smart in the book-learning sense, the very best at research will be far better than the mediocre.
  • To evaluate the trajectories of experts, consider their track record and incentives, try to correct for the Matthew Effect, and remember that transfer between paradigms is hard.
  • Relying on a small number of expert individuals to assess new contributions without the test of time has substantial limitations.
  • There are several possible paths that researchers believe will lead to AGI, including better environments, a neurosymbolic paradigm shift, bootstrapping through math or programming, scaling, and building better upstream representations.
  • Safety broadly construed is a very small part of the ML community, and most ML researchers do not like to talk about safety.

In the next post, we will discuss tactics for impact in AI safety research.



Discuss

Introduction to Pragmatic AI Safety [Pragmatic AI Safety #1]

9 мая, 2022 - 20:06
Published on May 9, 2022 5:06 PM GMT

This is the introduction to a sequence of posts that describe our models for Pragmatic AI Safety. Thanks to Oliver Zhang, Mantas Mazeika, Scott Emmons, Neel Nanda, Cameron Berg, and Michael Chen for feedback on this sequence.

Machine learning has been outpacing safety. Ten years ago, AlexNet pushed the boundaries of machine learning, and it was trained using only two GPUs. Now state-of-the-art models are trained on thousands of GPUs. GPT-2 was released only around three years ago, and today, we have models capable of answering bar exam questions, writing code, and explaining jokes.

Meanwhile, existing approaches to AI safety have not seen similar strides. Many older approaches are still pre-paradigmatic, uncertain about what concrete research directions should be pursued and still aiming to get their bearings. Centered on math and theory, this research focuses on studying strictly futuristic risks that result from potential systems. Unfortunately, not much progress has been made, and deep learning resists the precise and universal mathematical characterizations preferred by some safety approaches.

Recently, some established safety teams have focused more on safety in the context of deep learning systems, which has the benefit of being more concrete and having faster experimental feedback loops. However, many approaches often exhibit the downside of blurring the lines between general capabilities research and safety, as there appear to be few other options.

Finally, neither the pre-paradigmatic nor industry deep learning-based approaches seriously emphasize the broad range of sociotechnical factors that are critical for reducing risk from AI systems.

Given that ML is progressing quickly, that pre-paradigmatic research is not highly scalable to many researchers, and that safety research that advances capabilities is not safely scalable to a broader research community, we suggest an approach that some of us have been developing in academia over the past several years. We propose a simple, underrated, and complementary research paradigm, which we call Pragmatic AI Safety (PAIS). By complementary, we mean that we intend for it to stand alongside current approaches, rather than replace them.

Pragmatic AI Safety rests on three essential pillars:

  • ML research precedents. Safety involves technical AI problems, and the ML community’s precedents enable it to be unusually effective at solving technical AI problems.
  • Minimal capabilities externalities. Safety research at scale needs to be precautious and avoid advancing capabilities in the name of safety.
  • Sociotechnical systems view. Preventing catastrophes requires more than technical work, such as improving incentives, safety culture, protocols, and so on.
ML Research Precedents

Despite relying on “broken” processes like conferences and citations, the ML community has managed to solve an increasingly general set of problems: colorizing images, protein folding, superhuman poker, art generation, etc. This doesn’t mean that the ML community is set up optimally (we will discuss ways in which it’s not), but it does consistently exceed our expectations and demonstrate the best track record in solving technical AI problems.

In general, ML researchers are skilled at adding arbitrary features to systems to improve capabilities, and many aspects of safety could be operationalized so as to be similarly improved. This property makes ML research precedents promising for solving technical ML problems, including many safety problems.

Here are some ML research precedents that we view as important:

  • Long term goals are broken down into empirical simplified microcosmic problems
  • Subproblems can be worked on iteratively, collectively, and scalably
  • Contributions are objectively measured
  • The set of research priorities is a portfolio
  • Researchers must convince anonymous reviewers of the value of their work
  • Highly competitive, pragmatic, no-nonsense culture
  • Long-run research track records are necessary for success

We will address all of these precedents throughout this sequence. They are not original to safety, and some safety researchers have been following these precedents for years in academia.

There are many problems that we consider to be included in the PAIS research umbrella that are essentially ML problems or have essential ML components: honest AI, power-averseness, implementing moral decision making, value clarification, adversarial robustness, anomaly detection, interpretable uncertainty, detection of emergent behavior, transparency, ML for cyberdefense, and ML for improved epistemics. We will cover each of these areas in depth later in the sequence.

Lastly, we should note that we consider PAIS to overlap with problems considered relevant in ML, but there are a very large number of ML problems that are not relevant to PAIS (privacy, non-convex optimization, etc.). 

Minimal Capabilities ExternalitiesAn example of capabilities externalities. The safety method makes improvements on a safety-relevant metric while avoiding progress in capabilities, while a capabilities method may improve safety simply by moving along the existing safety/capabilities curve and thus not produce much differential impact.

Safety and capabilities are intertwined, especially when researching deep learning systems. For example, training systems to have better world models can make them less likely to spawn unintended consequences, but also makes them more generally capable. Optimizers that can operate over longer time horizons could ensure models don’t take problematic irreversible actions, but also allow models to make more complex longer-term plans in general.

It is clearly possible to make progress on safety metrics without improving capabilities, and some safety researchers have been doing it for years. But it must be done carefully. To do this, we propose a general policy of minimizing capabilities externalities. To the extent possible, we should avoid increasing capabilities in the name of safety, since this is a highly risky strategy and is not safely scalable for a broader research community. We should instead let the broader ML community take care of general capabilities, and work on safety problems that are viable with current capabilities. Rather than intuit whether something is good for overall safety, empirical researchers who follow our approach should demonstrate via measurement that their contribution does not simultaneously improve general capabilities.

Sociotechnical Systems View

A current blindspot in many AI safety approaches is to ignore nonlinear causality. Asking “how does this research agenda directly reduce this specific risk?” is well-intentioned, but it filters out accounts that capture nonlinear causal structures. Unfortunately, direct analysis is not expressive enough to model many of the most important phenomena relevant to safety. Today’s interconnected systems often have nonlinear causality, including feedback loops, multiple causes, circular causation, self-reinforcing processes, butterfly effects, microscale-macroscale dynamics, and so on. There may also be emergent behavior in an overall system that cannot be attributed to any individual subcomponent.

Remote, indirect, and nonlinear causes are omitted from accounts that require linear causality. Contemporary hazard analysis, and complex systems theory in general, is aware of this deficiency, and seeks to correct it. A central takeaway from these analyses is that it is essential to consider the entire sociotechnical system when attempting to prevent failures. Rather than only focusing on the operating process (in this case, a particular AI system’s technical implementation), we need to focus on systemic factors like social pressures, regulations, and perhaps most importantly, safety culture. Safety culture is one reason why engaging the broader ML community (including Chinese ML researchers) is critical, and it is currently highly underemphasized.

An example sociotechnical systems view from Leveson 2012. Much of AI safety has focused on the operating process but this process is inextricably linked to many other processes that cannot be ignored.The Pragmatic AI Safety Sequence

In this sequence, we will describe a pragmatic approach for reducing existential risk from AI.

In the second post, which will be released alongside this post, we will present a bird’s eye view of the machine learning field. Where is ML research published? What is the relative size of different subfields? How can you evaluate the credibility or predictive power of ML professors and PhD students? Why are evaluation metrics important? What is creative destruction? We will also discuss historical progress in different subfields within ML and paths and timelines towards AGI.

The third post will build on understanding from the first two posts and cover tactics for impact in AI safety. We will cover complex systems, benchmarks, systemic factors, portfolioization, problems with certain types of asymptotic reasoning, general heuristics for impact, and more.

The fourth post will serve as a supplement to Unsolved Problems in ML Safety. Unlike that paper, we will explicitly discuss the existential risk motivations behind each of the areas we advocate.

The fifth and final post will focus on tips for how to conduct good research and navigate the research landscape.

A supplement to this sequence is X-Risk Analysis for AI Research.

About the authors

This sequence is being written by Thomas Woodside and Dan Hendrycks as the result of a series of conversations they’ve had over the last several months.

Dan Hendrycks was motivated to work exceptionally hard after reading Shelly Kagan’s The Limits of Morality in high school. After leaving fundamentalist rural Missouri to go to college, he was advised by Bastian Stern (now at Open Philanthropy) to get into AI to reduce x-risk, and so settled on this rather than proprietary trading for earning to give. He did his undergrad in computer science; he worked on generic capabilities for a week to secure research autonomy (during which time he created the GELU activation), and then immediately shifted into safety-relevant research. He later began his PhD in computer science at UC Berkeley. For his PhD he decided to focus on deep learning rather than reinforcement learning, which most of the safety community was focused on at the time. Since then he’s worked on research that defines problems and measures properties relevant for reliability and alignment. He is currently a fourth and final-year PhD student at UC Berkeley.

Thomas Woodside is a third-year undergraduate at Yale, studying computer science. Thomas did ML research and engineering at a startup and NASA before being introduced to AI safety through effective altruism. He then interned at the Center for Human-Compatible AI, working on safety at the intersection of NLP and RL. He is currently taking leave from school to work with Dan on AI safety, including working on power-seeking AI and writing this sequence.



Discuss

Jobs: Help scale up LM alignment research at NYU

9 мая, 2022 - 17:12
Published on May 9, 2022 2:12 PM GMT

NYU is hiring alignment-interested researchers!

  • I'll be working on setting up a CHAI-inspired research center at NYU that'll focus on empirical alignment work, primarily on large language models. I'm looking for researchers to join and help set it up.
  • The alignment group at NYU is still small, but should be growing quickly over the next year. I'll also be less of a hands-on mentor next year than in future years, because I'll simultaneously be doing visiting position at Anthropic. So, for the first few hires, I'm looking for people who are relatively independent, and have some track record of doing alignment-relevant work.
  • That said, I'm not necessarily looking for a lot of experience, as long as you think you're in a position to work productively on some relevant topic with a few peer collaborators. For the pre-PhD position, a few thoughtful forum posts or research write-ups can be a sufficient qualification. We're looking for ML experimental skills and/or conceptual alignment skills/knowledge that could be relevant to empirical work, not necessarily both.
  • Our initial funding is coming from Open Philanthropy, for a starting project inspired by AI Safety Via Debate. Very early results from us (and a generally-encouraging note from Paul C.) here.
  • Pay and benefits really are negotiable, and we're willing to match industry offers if there's a great fit. Don't let it stop you from applying.


Discuss

Microphone on Electric Mandolin

9 мая, 2022 - 17:00
Published on May 9, 2022 2:00 PM GMT

A large portion of what I'm doing when I play rhythm mandolin is percussive: the sound of the pick hitting the strings. After listening back to the comparison between my electric and acoustic mandolins, this is the thing I find most lacking in the sound of the electric. What's interesting, however, is that I don't really notice this when playing at home. I realized a few days ago that it's hearing the electric mandolin acoustically that makes the difference, because the pickups are not really capturing these pick sounds. And if I can hear it acoustically, I can fix it with a microphone!

I made some recordings, and I really like the sound I get when I mix a little external microphone into the main sound from the pickup. I also really like that this means when I run the pickups through distortion or a talkbox the crisp sound of the percussion is unaffected.

Here's what it sounds like, recorded on an AudioBox VSL1818. The clip-on mic is an Audio-Technica Pro-35. In cases where I recorded the same thing last time, I've included the acoustic version for comparison. I've also included EvanY's EQ-matched versions from last time.

Overall, I feel like this is a huge improvement for chords and other percussive things, while it doesn't have much of an impact on melody either way.

Chords ( pickup only mp3)

( dual mp3)

( acoustic mp3)

( Logic EQ-matched mp3)

( Pro-Q EQ-matched mp3)

More Chords ( pickup only mp3)

( dual mp3)

Percussive ( pickup only mp3)

( dual mp3)

High Riff ( pickup only mp3)

( dual mp3)

( acoustic mp3)

Low Riff ( pickup only mp3)

( dual mp3)

( acoustic mp3)

High Melody ( pickup only mp3)

( dual mp3)

( acoustic mp3)

Medium Melody ( pickup only mp3)

( dual mp3)

( acoustic mp3)

Low Melody ( pickup only mp3)

( dual mp3)

( acoustic mp3)

I'm happy with this, and while this doesn't get the sound of the electric to where I would leave the acoustic behind on a Free Raisins gig, I think it will help Kingfisher mandolin sets a lot.

Comment via: facebook



Discuss

Thought experiment: Imagine you were assigned to help a random person in your community become as peaceful and joyful as the most peaceful and joyful person you'd ever met. What would you try?

9 мая, 2022 - 16:53
Published on May 9, 2022 1:53 PM GMT

What general methods, tools, approaches might you take?

Certain kinds of therapy, certain kinds of meditation, certain types of medication, certain types of coaching, certain activities, certain processes, and so on. Specific things, and/or meta things.

For constraints, imagine that you can spend 5 or 6 figures of US dollars per month, you're there in person full time with them. You don't know who it's going to be, but you know they don't have any notable medical conditions or past history. They're a semi willing participant. You can try anything that exists as of May 2022. 



Discuss

Willing to be your music mentor in exchange for video editing mentorship

9 мая, 2022 - 14:57
Published on May 9, 2022 11:57 AM GMT

Hey, I’m looking to trade expertise! I’m knowledgable in music theory, composition, improvisation, and music practice. I’m most proficient in guitar but I could teach piano as well.

I want to learn everything I can about video editing and would like a mentor. Here’s a list of some stuff I already know I need to learn about video editing: what it is, what tools I need, how to improve my taste, how to do it well, etc.

Open to having a video chat with anyone, don’t hesitate to ping me :)



Discuss

Updating Utility Functions

9 мая, 2022 - 12:44
Published on May 9, 2022 9:44 AM GMT

This post will be about AIs that “refine” their utility function over time, and how it might be possible to construct such systems without giving them undesirable properties. The discussion relates to corrigibilityvalue learning, and (to a lesser extent) wireheading.

We (Joar Skalse and Justin Shovelain) have spent some time discussing this topic, and we have gained a few new insights we wish to share. The aim of this post is to be a brief but explanatory summary of those insights. We will provide some motivating intuitions, a problem statement, and a possible partial solution to the problem given in the problem statement. We do not have a complete technical solution to the problem, but one could perhaps be built on this partial solution.

Sections which can be skipped are marked with an asterisk (*).
 

Brief Background*

This section says things that you probably already know. The main purpose of it is to prime you.

In the “classical” picture of AI systems, the AI contains a utility function that encodes a goal that it is trying to accomplish. The AI then selects actions whose outcome it expects will yield high utility (roughly). For example, the utility function might be equal to the number of paperclips in existence, in which case the AI would try to take actions that result in many paperclips. 

In the “classical” picture, the utility function is fixed over time, and corresponds to an equation that at some point is typed into the AI’s source code. Unfortunately, we humans don’t really know what we want, so we cannot provide such an equation. If we try to propose a specific utility function directly, we typically get a function that would result in catastrophic consequences if it were pursued with arbitrary competence. This is worrying.

This problem could perhaps be alleviated if we could construct AIs that can refine their utility function over time. For example, maybe we could create an AI that starts out with an imperfect understanding of human values, but then improves that understanding over time. Such an AI should ideally “want” to improve its understanding of human values (and actively come up with ways to do this), and it should at minimum not resist if humans attempt to update it. Unfortunately, it turns out to be difficult to design such systems. In this post we will talk more about this approach.

 

A Puzzle of Reference*

Consider this puzzle: I am able to talk and reason about ”human values”. However, I cannot define human values, or give you a definite description of what human values are – if I could do this, I could solve a large part of the AI alignment problem by writing down a safe utility function directly. I can also not give you a method for finding out what human values are – if I could do this, I could solve the problem of Inverse Reinforcement Learning. Moreover, I don’t think I could reliably recognize human values either – if you show me a bunch of utility functions, I might not be able to tell if any of them encodes human values. I’m not even sure if I could reliably recognize methods for finding out what human values are – if you show me a proposal for how to do Inverse Reinforcement Learning, I might not be able to tell whether the method truly learns human values.

In spite of all this, the term “human values” means something when I say it – it has semantic content, and refers to some (abstract) object. How does this work? What makes it so that the term “human values” even has any meaning at all when I say it? And, given that it has a meaning, what makes it so that it has the particular meaning it does? It seems like some feature of human cognition and/or language can make it possible for us to refer to certain things that we have very little information about. What is the mechanism behind this, and could it be used when defining utility functions in AI systems?
 

Problem Statement

We want a method for creating agents that update their utility function over time. That is, we want:

  1. A method for “pointing to” a utility function (such as “human values”) indirectly, without giving an explicit statement of the utility function in question.
  2. A method for “clarifying” a utility function specified with the method given in (1), so that you in the limit of infinite information obtain an explicit/concrete utility function.
  3. A method for creating an agent that uses an indirectly specified utility function, such that:
    • The agent at any given time takes actions which are sensible given its current beliefs about its utility function.
    • The agent will try to find information that would help it to clarify it’s utility function.
    • The agent would resist attempts to change its utility function away from its indirectly specified utility function.

This problem statement is of course somewhat loose, but that is by necessity, since we don’t yet have a clear idea of what it really means to define utility functions “indirectly” (in the sense we are interested in here).
 

Utility Functions and Intensional Semantics*

What is in this section is a tangent about wireheading -- it might be interesting to read this while thinking about this topic, but it is not necessary to do so.

How should an AI evaluate plans if its utility function changes over time? Suppose we have an AI that currently has utility function U1, and that it considers a plan P that would lead to outcome O, where in O the AI would have the utility function U2. Should the utility of P be defined as U1(O) or U2(O)? If it’s U1(O) then the AI is maximizing its utility function de re, and if it’s U2(O) then it’s maximizing its utility function de dicto. Which is more sensible?

In brief, an AI that maximizes utility de re will resist attempts to modify its current utility function, and thus not satisfy (3). An AI that maximizes utility de dicto would wirehead, and thus also not satisfy (3). An AI that maximizes utility de re would not wirehead.

This is perhaps a somewhat interesting observation, but it doesn’t help us solve (1)-(3).
 

Limiting Utility Functions -- Possibly a Partial Solution

Let’s  define a process P that generates a sequence of utility functions {Ui}. We call this a utility function defining process. An example of such a process P could be the following:

P is an episodic process, the input and output to which is one proposed human utility function and one set of notes. Given these, P runs n human brain emulations (EMs) forsubjective years. The brains can speak with each other, and have a copy of the internet that they can access. The EMs are meant to use this time to figure out what human preferences are. At the end of the episode they output their best guess, together with a set of notes for their successors to read. By chaining P to itself we obtain a sequence of utility functions {Ui}.

We would like to stress that this process P is an example, and not the central point of this post.

Suppose (for the sake of the argument) that the sequence of utility functions {Ui} generated by this process P has a well-defined limit U∞ (in the ordinary mathematical sense of a limit). We can then define an AI system whose utility function is to maximize lim i→∞ Ui (= U∞). It seems as though such a system would satisfy many of the properties in (1)-(3). In particular:

  • The AI should at any given time take actions that are good according to most of the plausible values of U∞.
  • The AI would be incentivized to gather information that would help it learn more about U∞.
  • The AI would not be incentivized to gather information about U∞ at the expense of maximizing U∞ (eg, it would not be incentivized to run “unethical experiments”). 
  • The AI would be incentivized to resist changes to its utility function that would mean that it’s no longer aiming to maximize U∞.  
  • The AI should be keen to maintain option value as it learns more about U∞, until it’s very confident about what U∞ looks like.

Overall, it seems like such an AI would satisfy most of the properties we would want an AI with an updating utility function to have.

To clarify, note that we are not saying that you run the utility function defining process P to convergence and then write the utility function you end up with into the AI – you would not need to run P at all. The purpose of P is to point to U∞ – the work of actually finding out what U∞ is is offloaded onto the AI. The AI might of course do this by actually running P, but if P is very complex (as in the example above) then the AI could also use other methods for gaining information about U∞.

Again, we stress that the point here isn’t the specific process P we propose above – that is just an example. As far as the approach is concerned, you could use any well-defined process that produces a sequence of utility functions that converges to a well-defined limit.
 

Issues

There are a few issues with this approach. Notably:

  1. The approach is very unwieldy, and it seems like it requires a fairly high minimum level of intelligence to work. For example, it couldn’t be used as-is with a contemporary RL agent. 
    • It’s not clear what would be needed to use this approach with an AI that starts out below the minimum required level of intelligence, but then gets more intelligent over time.
    • The nitty-gritty details of getting an AI system to maximize the limit of a mathematical sequence would in general presumably require good methods for dealing with logical uncertainty.
  2. We still need to provide a specific process P, such that we are sure that P has a well-defined limit, and such we are confident that this limit corresponds to the utility function that we are actually interested in.
    • Note however that this might be much easier than, for example, solving Inverse Reinforcement Learning. For example, there isn’t really any need for P to be efficient or practical to run.
  3. With the current version of this approach, all the information required to figure out what U∞ is must in some sense be contained within P from the start. This is problematic – what if it’s not possible to figure out what human values are based on all information that can be accessed when the system is deployed? For example, what if you need some facts about the human brain that just aren’t in the scientific literature at the time?
    • One way to get around this is to allow P to request new external information (by proposing an experiment to run, for example). However, this introduces new difficulties. Depending on what information is requested, this could make the value of U∞ depend on contingencies in the real world. In particular, it could make the value of U∞ depend on things that the AI can influence. For example, if P requests that a survey is run then the AI could probably influence the outcome of that survey (and the outcome would also depend on the specific time at which the survey is run, etc etc). In this case it’s unclear how you would even ensure that U∞ is well-defined, and it seems very difficult to ensure that the AI still has the intended incentives.

Nonetheless, it seems like this approach has many nice and desirable properties, and the issues are not fatal, so it might still be possible to use this approach in an AI system, or build on it to create an even better approach.
 

Conclusion

In summary, we want a method for pointing to utility functions that works even if we don’t have a concrete expression of that function (like how I can point to human values by saying “human values”, even though I can’t say much about them). We also want a method for making an AI system maximize a function that has been pointed to in this way, which doesn’t incentivize bad behavior.

We have proposed a possible approach for doing this, which is to define a mathematical or computational process that generates a sequence of utility functions, which limits to some well-defined utility function, and then have the AI system try to maximize that limit function. This gives us a quite flexible way to define utility functions, and the resulting AI system seems to get the incentives we would want.

This approach has a few limitations. The most problematic of these is probably that it seems to induce a fairly large overhead cost, in terms of computational complexity, in terms of the complexity of the code, and in terms of how intelligent the AI system would have to be. Other issues include defining the utility function generating process, ensuring that it has a well-defined limit, and ensuring that that limit is the function we intend. However, these issues are probably less significant by comparison, since other methods for defining AGI utility functions usually have similar issues.

 

The prompting idea for this post was from Justin Shovelain, Joar Skalse and Justin Shovelain collaboratively came up with the much improved Updating Utility Functions idea, and Joar Skalse was the primary writer.



Discuss

Transcripts of interviews with AI researchers

9 мая, 2022 - 08:57
Published on May 9, 2022 5:57 AM GMT

tldr: I conducted a series of interviews with 11 AI reseachers to discuss AI safety, which are located here: TRANSCRIPTION LINK. If you are interested in doing outreach with AI researchers, I highly recommend taking a look!

 [Cross-posted to the EA Forum.]

Overview

I recently conducted a series of interviews with 11 AI researchers, wherein I laid out some reasons to be concerned about long-term risks from AI.

These semi-structured interviews were 40-60 minutes long and conducted on Zoom. Interviewees were cold-emailed, were paid for their participation, and agreed that I may share their anonymized transcripts.

Six of the interviews were with researchers who had papers accepted at NeurIPS or ICML in 2021. Five of the interviews were with researchers who were informally categorized as “particularly useful to talk to about their opinions about safety” (generally more senior researchers at specific organizations).  

I’m attaching the raw transcripts from these 11 interviews, at the following link. I’ve also included the approximate script I was following, post-interview resources I sent to interviews, and informal interview notes in the associated “README” doc. Ideally I’d have some analysis too, and hopefully will in the future. However, I think it’s useful— particularly for people who plan to start similar projects— to read through a couple of these interviews, to get an intuitive feel for what conversations with established AI researchers can feel like.

Note: I also interviewed 86 researchers for a more complete academic, under-IRB study (whose transcripts won’t be released publicly), whose results will be posted about separately on LessWrong once I finish analyzing the data. There will be substantially more analysis and details in that release; this is just to get some transcripts out quickly. As such, I won't be replying to a lot of requests for details here.

Thanks to Sam Huang, Angelica Belo, and Kitt Morjanova, who helped clean up the transcripts! 

Personal notes 
  • I worry that sometimes young EAs think of researchers like a crowd of young, influenceable proto-EAs. One of my models about community-building in general is that there’s many types of people, some who will be markedly more sympathetic to AI safety arguments than others, and saying the same things that would convince an EA to someone whose values don’t align will not be fruitful. A second model is that older people who are established in their careers will have more formalized world models and will be more resistance to change. This means that changing one’s mind requires much more of a dialogue and integration of ideas into a world model than with younger people. The thing I want to say overall: I think changing minds takes more careful, individual-focused or individual-type-focused effort than would be expected initially.
  • I think one’s attitude as an interviewer matters a lot for outcomes. Like in therapy, which is also about changing beliefs and behaviors, I think the relationship between the two people substantially influences openness to discussion, separate from the persuasiveness of the arguments. I also suspect interviewers might have to be decently “in-group” to have these conversations with interviewees. However, I expect that that in-group-ness could take many forms: college students working under a professor in their school (I hear this works for the AltProtein space), graduate students (faculty frequently do report their research being guided by their graduate students) or colleagues. In any case, I think the following probably helped my case as an interviewer: I typically come across as noticeably friendly (also AFAB), decently-versed in AI and safety arguments, and with status markers. (Though this was not a university-associated project, I’m a postdoc at Stanford who did some AI work at UC Berkeley). 
  • I’d be very excited if someone wants to take my place and continue on this project with one-on-one discussions with researchers. There’s a good chance I won’t have time to do this anymore, but I’m still very enthusiastic about this project. If you think you’d be a good fit, please reach out (vaelgates@gmail.com)! Things that help: friendliness, as much familiarity with AI arguments as you can manage (I also learned a lot about gaps in my models through these interviews), conscientiousness, and some way to get access to researchers. Extroversion would have helped me a lot too, but can’t have everything.

                                                       TRANSCRIPTION LINK



Discuss

[Scribble] Bad Reasons Behind Different Systems and a Story with No Good Moral

9 мая, 2022 - 08:21
Published on May 9, 2022 5:21 AM GMT

I wrote this on impulse as a response to “A Reason Behind Bad Systems, and Moral Implications of Seeing This Reason”. It is essentially a first draft and is probably badly written. The plot also got away from me in the later sections; my mental processes surrounding a lot of these phenomena are too cross-linked. However, I sense it may be more useful for this to exist in the open than not, and I think if I decide to hide this away for rewriting, realistically I will not do so and it will languish forever… so here it is. Perhaps it can become something better later; perhaps not.

For copyright and moral rights purposes, as the sole author of everything in this post that is not a quotation, I place all such text in the public domain.

Epistemic status: fictional, quasi-allegorical, biased, exploratory.

I

The programmer with the messy program did what any sensible software engineer would have done in her shoes: Set aside the old software and built something useful, that did what was needed, using the full power of the company’s modern computers.

A month later, Evelyn's manager at Yottascape, Henrietta, stormed into Evelyn's office with a notepad about the hundred different telephone calls the department had gotten complaining that the results from the new system were subtly inconsistent with those from the old. Other people who had little to no understanding of the code behind the figures they were seeing had built processes and expectations based on patterns they observed, patterns which had stayed stable for years.

Some of the new outputs had caused cascading instabilities. One manufacturing location had run out of slack on a key inflow of materials, temporarily halting outflow of parts to a second location, which revealed that workers at a second location had become lax in ensuring that the machine they were using—which needed to be kept at a certain temperature or suffer expensive damage that would put it out of commission for months—was actually kept at that temperature. The intake at the second location had been so constant before that it created a self-regulating effect, and they had stepped down control checks to one third the original frequency, a cost-saving measure which had been widely praised by management. Now the damage would lead to broken promises which would have to be documented in their next SEC filing. Even before then, it would involve a flurry of contract renegotiations with distributors, investors, and suppliers, some of whom would gossip among themselves, sensing that something was amiss and that their common counterparty had fallen off-balance, and push ruthlessly for favorable terms that would strip Yottascape to the bone.

Evelyn pointed out that her new outputs were, in fact, more correct than the old ones on several axes, and that she had documented these in the white paper she had sent out ahead of time. Hadn't they read her equations? The new code would also allow for much faster evolution, and it would be much easier to understand. She had projected that it would save a million dollars a year in maintenance costs, and potentially another million dollars a year in efficiency improvements. And her projections about software maintenance had always been correct before.

Henrietta replied that since Evelyn was a senior in the department who was given great operational leeway, it was part of her job description to ensure she got appropriate feedback and that the effects were properly understood before implementing such a drastic change unilaterally. Additionally, the finance department was projecting several million dollars a year of reduced revenue for several years yet as a result of the fault cascade. Further, less easily quantifiable losses might well accrue from morale loss leading to key innovative workers in the field leaving when their own efforts turned out to be wasted as a result of unpredictable events like this.

Evelyn was fired. Her code was rolled back, and the second most senior in the department, Saul, was promoted to lead the adjustment project in her stead. Saul gave an expansive presentation to Evelyn about how he would ensure his new code would not disrupt operations, and how there would be an extensive network of backup plans, rollback options, and safety checks, as well as additional advance training for anyone who might be affected by changes in this most critical program. He also noted that he would need a budget and access to managerial authority sufficient to take the time out of the workdays of other affected parties to ensure this would go smoothly. Henrietta worked with the executives above her and got all of this approved, despite wincing at the additional cost.

Six months after the original incident, gradual uptake was proceeding mostly apace—well, actually, it was notably slower than expected, as it seems Saul had been somewhat optimistic—but no further massive disruptions had accrued. Most of the acute damage had since been repaired, and while everyone still suffered from the effects of the more onerous contract terms and did not reasonably expect them to end soon, they had gotten used to it. Things had returned, mostly, to normal, with the prospect of new, safer improvements on the horizon.

II

Twelve months after the original incident, a hotshot new company, Zettaboom, announced product lines competing with three of Yottascape's main revenue sources. They were already delivering with more options and higher reliability at half the price, and the tide of consumer sentiment, including the lingering (if tarnished among anyone who read the business section of the newspaper) perception of Yottascape as the premier manufacturer in their space, was starting to turn away. Henrietta frantically checked to see whether Zettaboom had hired Evelyn—and they hadn't. But they had contracted with a consultancy that Evelyn had joined. The Zettaboom product managers were careful to keep the consultants' work at arm's length and retain complete control over acceptance criteria, but they also remained open-minded and lacked the burden of legacy procedure, and the code quality from Evelyn and her new coworkers was a key factor in their ability to pivot quickly and handle new market conditions while keeping costs and risks down. This appealed greatly to Zettaboom's investors, allowing them fast access to surprising amounts of capital to grow as a Yottascape competitor.

III

Five years after the original incident, unrest was rising among the public. Life had grown too hectic, somehow. Everything was shinier and glossier on the surface, but they all felt worse off. Those who could adapt to chaos well had gravitated to key positions in society, and then the chaos itself had become an evolutionary defensive measure, one not imposed by any dictat but only by the jitter of tense, barely-held-together, complex, implicit control systems. Old pillars had crumbled, somehow, from the weight of compounded individual choices. No one had consciously decided that Yottascape's old-fashioned community involvement should be diminished, or noticed the subtle differences in worker lifestyle, only made decisions using the information they had, and gotten a wave of recommendations from friends and acquaintances and ‘known’ that this was where society was going and they'd better get with the times.

Some people blamed everyone including themselves for getting in the habit of buying from Zettaboom and similar ‘soulless’ corporations, but very few had anything like a solution to propose, and no one could deny that the benefits had existed too—they could only lament that they hadn't coordinated on different priorities earlier. Others pointed out that executing on those priorities the way the first set of complainants wished had happened might not have worked out either. No one from either group could make any headway; the cultural and structural drift had made it almost impossible for someone of median intelligence or median social connections to make it very far in the world, creating a semi-permanent class rift.

Those of great mental capacity who did make it into the managerial class immediately entered a cycle of having to prove themselves by wielding all their intelligence to create more and more complex systems. Anyone who tried to simplify things would be seen as weak, and their market positions would erode, their influence would be treated with disgust, and they would quickly be expelled to become part of the underclass, below even the commoners. If they tried to rejoin the commoners, after all, they would be viewed with a different disgust as having contributed to The Problem, and they would have already rendered themselves unfit in habit to take part in any of it—so they were left with nowhere to go. Knowing that this could happen made those of the managerial class even more anxious.



Discuss

What is the best day to celebrate Smallpox Eradication Day?

9 мая, 2022 - 07:02
Published on May 9, 2022 4:02 AM GMT

Several of my friends posted on Facebook/threw parties this weekend for a date of May 8. However, searching on Facebook, I find that I and several other friends have been posting celebratory messages on Facebook on December 9 in previous years.

This inconsistency is probably because there are two obvious candidate dates for eradication:

The global eradication of smallpox was certified, based on intense verification activities, by a commission of eminent scientists on 9 December 1979 and subsequently endorsed by the World Health Assembly on 8 May 1980.

"500 Million, But Not a Single One More", which is widely shared and has been adapted at least once for reading at a large Secular Solstice, uses the December 9 date.

 

I think creating more consensus on which date is the "official" one for celebration purposes would be a small improvement (at the very least: I would stop confusing my relatives by year-to-year inconsistency on when I message them a little commemoration). So, which is it?



Discuss

A REASON BEHIND BAD SYSTEMS, AND MORAL IMPLICATIONS OF SEEING THIS REASON

9 мая, 2022 - 06:16
Published on May 9, 2022 3:16 AM GMT

(Epistemic status: Pretty sure the premises are all correct and the metaphor is accurate. Exploring the implications and conclusions.)

A programmer was given an aging piece of software to use to complete a project. To the company, using the old software seemed like the path of least resistance. But, opening the program and looking at the code, the programmer saw it had been hacked, edited, and added to by twenty or more programmers before her, each with their own style and vision of what a good program should be, each according to the conventions and concerns of the day.

"Ahhh, Legacy code," the programmer said. She frowned a little.

By now, the software was full of unused code and had a maze of lines of programming that referred to other lines that referred to yet other lines, and so forth. The programmer found it almost impossible to sort out exactly what many parts of the program were designed to do. Worse, though the computers used by the company were high-tech, parts of the software were designed to run on computers from decades before. The program was a nightmare to work with. Normally one had to doctor the finicky inputs, and often the software returned incorrect, puzzling, or even meaningless outputs.

When software like this does anything useful at all, it is a small miracle.

Most organizations, social hierarchies, and belief systems are heavy with garbage. There is no one to blame, because most of those systems are constructed from layers of intentions and interpretations spanning years, decades, or even centuries. Hence things like a legal system with thousands of pages of code and case law. Plenty of these systems evolved around social realities that have changed immensely since they were designed. Meanwhile, your own experience tells you that no matter how dysfunctional or downright it is, no organization, belief system, or social structure will ever agree with you that it is wrong. For the most part, none of these systems were designed to take the steps to eliminate themselves when they are no longer useful.

Once a system takes root in society and ages, it loses its agency, intelligence, and will. That system is usually very hard to remove even if it starts to work against its original intention. Most people either thoughtlessly adopt or begrudgingly embed themselves in whatever systems are presented by their culture and society. Doing so usually seems to be the path of least resistance, even necessity. Thus, many social structures and systems are aging and barely functioning software, legacy programs running on human hardware.

The programmer with the messy program did what any sensible software engineer would have done in her shoes: Set aside the old software and built something useful, that did what was needed, using the full power of the company’s modern computers.

REGARDING MORAL OBLIGATIONS TO SYSTEMS:

No one in history has ever died wishing they had paid more dues to hierarchies, bureaucrats, and society’s systems. Yet that is what those systems always seem to want: More. People treat them as basic truths, but do we owe such systems any more loyalty than we would give to old computer programs?

The next time old software gets hung up on a certain procedure unless you give it doctored inputs it likes to see, why even hesitate? When you recognize opportunities to bypass or delete such software, should your default choice be to seize them?

And finally, the clear implication is we should be writing new software, and attempts to "tweak" it are probably just going to pile on more garbage and spin it further out of spec. I think Ed Deming would advocate strongly that this is in fact the case. This could be an even stronger moral impetus to "delete/bypass" such a system.

I see the main conflict in my reasoning would be with people who have embedded themselves by default in the systems around them. It would be like all the people who accepted a bloated Windows because it's all their org and Best Buy ever gave them and now we're all switching to Linux. Maybe then the moral obligation is to try to facilitate "soft landings" for those already deep in the current systems.



Discuss

An Alternative Interpretation of Physics

9 мая, 2022 - 03:52
Published on May 9, 2022 12:52 AM GMT

 1. Physics With Minimal Assumptions

“I am this thing”. That is perhaps the most basic fact I know of. It is not a conclusion based on reasoning, but something much more instinctive: “I know how it feels.“. For me, the only subjective experience is from this thing.  It’s at the very center of my perspective. Let’s call it “self”.

I learn about my surrounding by interacting with them. Via these interactions, I formed the conception of external objects and the world around me. If I study the interactions carefully, I would discover certain patterns: rules that can explain and predict them. These rules also help me to describe external objects as the interactions’ counterparties.

Then there’s a crucial realization: the first-person perspective I’m experiencing doesn’t have to be the only valid perspective. I realized external objects also interact with their environments. I can imagine thinking from a particular object’s viewpoint. So that thing would become the “self”, and I become an external object interacting with it.

Back to the rules that my interactions seem to follow. There are restrictive rules, that only work for a limited range of perspectives and interactions. And there are other, more general rules applicable to a wide range of perspectives. I should try to induce rules of this type as they give a deeper insight into how the world works. Let’s call the general rules “physical laws”, and the depictions of the world using these laws “physical descriptions”.

2. Classical Physics Is Weird

This parsimonious interpretation of physics has some implications. First of all, physical analysis has to be conducted from a prespecified perspective. It can be the natural first-person perspective of you or me or the perspective of any object. Which perspective is the reasoning to be conducted from i.e. what is the “self”, is an exogenous input, not something to be explained by physics.

Following the above, studying interactions received by the “self” (the thing at the perspective center) enables conceptualizing external objects as the actions’ counterparties. Physical descriptions of external objects are based on these interactions.

Finally, because objects are described based on their interactions with the self, the “self” itself is not describable.  I.E. the “thing at the perspective center” is not within the domain of explanation. To physically analyze something we must not take the perspective of that object, but the perspective of some other things interacting with it.

These implications fit very nicely with the interpretive challenges of quantum mechanics. For example, the special role of the “observer” is to be expected. It is simply the prespecified self, from whose perspective the analysis is conducted. “Measurement” is not a convoluted notion either, just an interaction with the “self”. Physically describing something not interacting with the observer is impossible.

With this approach, quantum mechanics seems unsurprising. Instead, the interesting question is how come classical physics can work by thinking directly in terms of an absolute “physical reality” without paying any attention to perspectives?

3. Energy Saving Approximation

Directly thinking in terms of the “absolute reality” is an intuition formed by living in the macroscopic world. In our daily life, we only conceptualize large objects and do not make precise physical descriptions.

Imagine there is a glass bottle in front of you. A camera is also pointing at it. From your perspective, some of your interaction with the environment is, directly or indirectly, affected by that bottle. Same for the camera’s perspective. Therefore the bottle can be conceptualized from either perspective by studying the interactions. Yet, even using the same physical laws, the descriptions from the two perspectives are inevitably going to be different since they are based on different sets of interactions.

However, for a coarse-grained characterization, the detail differences are swept away. For example, the photons get in my eyes bounced off from different atoms of the bottle than those photons get into the camera, therefore the two perspectives won’t give identical descriptions at the atomic level. But if we are only interested in macro-level features such as “the shape of the bottle”, then both perspectives have countless interactions giving enough information well beyond the need to deliver these rough depictions. Their description of the macro features would be virtually identical.

Furthermore, because daily objects are massive, there would be numerous interactions for any practical choice of perspectives. The interactions are effectively continuous so the objects can be described at any given moment. Effectively, for coarse-grained macro-level physical analysis, there are two important traits: perspective invariability, and object permanence. This gives rise to the idea of the “absolute physical reality” that we can reason directly about. Without needing to define which perspective is the analysis conducted from.

And we have good reason to do so. To correctly capture the descriptive differences across perspectives requires analyzing minuscule interactions highly accurately. That is perceptively and computationally very expensive yet offers marginal benefit for our survival. It is much more efficient to treat external objects as absolute fundamental existences rather than derived conceptualizations.

Nonetheless, thinking in terms of this “absolute reality” is just a useful approximation. The approximation breaks down gradually as we begin focusing on the microscopic. Where interactions are few and far between and a high level of accuracy is required.

4. The God’s Eye View and Schrodinger’s Box

Some may consider the classical way of thinking in terms of absolute reality as merely conducting the analysis from a god’s eye view. So it is no different from the perspective-based understanding of physics as laid out earlier. But that is not so.

There is nothing inherently wrong with thinking from an imaginary thing’s perspective. However, reasoning directly in terms of physical reality uses a view from nowhere. There is no self: nothing is at the perspective center. So everything is part of the external environment. Hence the domain of physical explanation includes the whole world. Furthermore, such a view is omniscient as it doesn’t need interactions to describe objects. It is an unjustified assumption to say physical laws applicable from the perspective of ordinary things can also be used from such a supernatural perspective.

However, due to intuitions formed in daily life, we tend to use the god’s eye view even when the approximation breaks down. Doing so can result in perplexing conclusions. Schrodinger’s cat is perhaps the best-known example.

Let’s assume the ideal box cuts off all interactions between the inside and the outside. For things inside the box, there are numerous interactions available to describe the cat. From such perspectives, the cat is clearly either dead or alive. For things outside the box, no interaction gives any information about what’s happening on the inside. From these perspectives, the current physical state of the cat is indescribable. The superposition of alive/death is not the absolute physical state of the cat from a god’s eye view. It is merely a prediction of what kind of interactions someone outside the box would expect.

If the experiment uses an ordinary box that blocks some but not all interactions, the conclusion would be different. In this case, the scale of objects does make a difference. Because a cat is massive and “alive or dead” are coarse-grained characterizations, it’s almost certain that there would be enough interactions carrying information necessary to make that description, even for someone outside but reasonably close to the box (or think from the perspective of “all things outside”). We might not analyze all the interactions to make the description. But the cat is in a definitive physically describable dead or alive state from such perspectives.

5. Some Metaphysical Stance

One thing that needs to be cleared is this interpretation does not deny the reality of the world or endorse solipsism. On the contrary, it takes the reality of the world as a primary postulate. It believes other things’ existence is as real as my own. Therefore their perspectives are as valid as mine. It simply suggests we have no way to reason about reality directly. The classical notion of “absolute physical reality” does not cut it. We are bounded to learn the true nature of the world through perspectives and interactions. They are the innate spectacles we cannot shake off.

It is also noteworthy that this interpretation puts physics and consciousness in an alternative reasoning framework. As discussed, physics cannot describe the thing from whose perspective the analysis is conducted. Its scope of explanation does not include the “self”. On the other hand, subjective experience and consciousness exclusively apply to the self.  (i.e. From my perspective, the only subjective experience that exists is that of my own.) So physics is released from the duty of explaining subjective feelings or closing the explanatory gap. As it is ontologically impossible.

Closely related to consciousness, free will is also a feature inherently applicable to the self. One can only contemplate choices or make decisions from a first-person perspective. In contrast, physically analyzing an external object to deduce its output does not involve the notion of free will at all. (This is actually the cause of Newcomb’s Problem as I have written earlier.) Due to this separation, physics cannot be used to deny the existence of free will.

Finally, since this interpretation does not consider consciousness as a physical feature but inherent to the prespecified perspective center, it is a form of panpsychism. It is not suggesting everything’s subjective experience feels similar to mine or yours. There is no way to know that.  But a thing can be considered conscious as long as we choose to reason from its perspective.



Discuss

Ion Implantation: Theory, Equipment, Process, Alternatives

9 мая, 2022 - 01:44
Published on May 8, 2022 10:30 PM GMT

 Ion implantation is a common process used in the semiconductor industry to change the properties of a material, namely silicon (the substrate). Physics, equipment used, process considerations, alternatives, and further resources are discussed.

Note: I copied this direct from my website and did not take the time to format the inline math. The symbols should be clear enough that subscripts/superscripts are not needed.

Contents
  • Physics
    • Dose vs. Concentration
    • Stopping
    • Activation Anneal
  • Equipment
    • Sources
    • Mass Analyzer
    • Acceleration
    • Endstation
  • Process Considerations
    • Implant Masks
    • Channeling
    • Uniformity
  • Doping Alternatives
    • Diffusion Doping
    • Spin-On Doping
  • Further Resources
  • See Also
Physics

First, why are ions shot into Si? What does it do from a physics perspective?

The reason has to do with the band gap, the energy range between the valence band and conduction band.

Within Si exist both electrons (negatively-charged) and electron hole (positively-charged). The hole is a bit of an oddity: it's an unoccupied space where an electron could exist, but doesn't at that moment. While not literally a particle, they can be treated and thought of as such. Current can be viewed as the flow of electrons or holes. When a bias is applied, electrons begin to move and occupy holes. When electron A moves to hole B, hole A is formed. This propagates and creates the flow of charge, or current.

What does this have to do with Si? Si has four valence electrons in its four states of 3s23p2. Those four valence electrons covalently bond with the surrounding Si crystal lattice structure, effectively filling that Si atom's valence shell. So, when an ion is introduced into the lattice that does not have four valence electrons, one of two things happen: there are free electrons (if the number of valence electrons is >4, such as in phosphorus's case) or free holes (valence electrons <4, such as boron). In the first case, four of P's electrons have bonded to surrounding Si atoms' valence electrons, leaving the fifth one free and able to contribute to current—a donor impurity, because it donates an electron. This creates an n(egative)-type semiconductor. In the second case, three of B's electrons have bonded to surrounding Si atoms, but one Si atom and the B atom both have incomplete valence shells (7/8, 6/8 filled, respectively). To fill the vacancy, the Si and B will pull a free electron from the lattice, leaving a hole in its place—an acceptor impurity, because it accepts an electron. This creates an p(ositive)-type semiconductor.

The Fermi energy level, EF, is the energy level at which the probability an electron exists there is 0.5, based on the Fermi-Dirac distribution:

f(E)=11+exp(E−EFkBT).mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}

EF for an intrinsic semiconductor (no impurities added) is given by:

EF=Ei=EC+EV2+kBT2ln(NVNC)

where Ei is the intrinsic Fermi level, a constant, and NC and NV are the effective density of states. (Note that the link only shows you density of states, g(E), not effective DOS. The effective DOS derivation can be found here.)

While Ei cannot change, EF can based on the number of ionized impurities present in the substrate. When an n-type dopant is added, EF shifts up towards the conduction band, decreasing the EC−EF gap. Similarly, when a p-type dopant is added, EF shifts down towards the valence band, decreasing the EF−EV gap. Because there are now electrons sitting at the new n-type EF level, even a small amount of thermal energy can excite them into the conduction band, creating current. For the p-type, electrons can also be excited to EF, leaving holes in the valence band, creating current.

It's this change in the EF value that drives modern-day devices (admittedly, some other things help, too). By combining two substrates—one n-type and one p-type—and creating a p-n junction, both of the EF levels must match up. The p-n junction is the basis for transistors, proving the importance of ion implantation in devices.

Dose vs. Concentration

Two common terms used are dose and concentration. They are similar, but not synonymous. Dose is number of ions per unit area (cm2), while concentration is number of ions per unit volume (cm3).

Given beam current, I (charge/time), and time spent implanting, t, the total amount of implanted charge is simply Q=It. Dose, ϕ (number/area), can be multiplied by the elementary charge, q, and scan area, A, to also get the total implanted charge. Integrating the current with respect to time can also calculate dose, which is actually how the tool itself calculates dose (see dose measurement). Equating the first two and solving for dose gives:

ϕ=ItqA=1qA∫I(t)dt

Dose can also be found be integrating the concentration, N(x), with respect to depth:

ϕ=∫N(x)dx

The concentration of an implant can be approximated if the range, Rp, and straggle, ΔRp, are known (range and straggle values are experimentally-determined and based on energy):

N(x)=N0exp(−(x−Rp)22ΔR2p)=ϕΔRp√2πexp(−(x−Rp)22ΔR2p)

The N(x) equation above is a simple Gaussian function, but is not entirely representative of how actual implant profiles look. To better simulate real profiles, the Pearson IV distribution and moments are used, which accommodate for the skewness that often occurs.

Boron distribution in Si at 30, 100, 300, 800 keV - the skewness becomes more obvious (source)

The respective moments are as follows:

  1. Rp: Average depth of an implanted ion from the surface of the substrate.
  2. ΔRp: Variance of the depth.
  3. γ: Skewness, or how much the profile leans to one side (on a distance x-axis, meaning towards or away from the surface). A distribution that is skewed left (long tail on left side) has negative skewness (higher concentration towards the surface), while a distribution that is skewed right has positive skewness (higher concentration deeper in the substrate). A perfect Gaussian distribution has no skewness because it is perfectly symmetrical.
  4. β: Kurtosis, or heaviness (or fatness) of the tails.

The integral limits are from 0 to ∞ because a negative number would not make sense when referring to depth. Using all of those, the Pearson equation can be solved and specific values substituted in to the IV solution. For specifics, see pages 8-12 of Dr. Alan Doolittle's lecture.

Range and straggle values as a function of energy for common dopant species (source)

A typical concentration vs. depth Gaussian curve is shown below:

Concentration vs. depth for a P+ implant through a SiO2 mask into a Si substrate (source)

The concentration profile makes intuitive sense, especially if you rotate the image 90° and superimpose the mask and substrate onto it:

Depth vs. concentration for a P+ implant with mask and substrate superimposed (source)

It is seen that the peak concentration occurs about halfway through the SiO2. The concentration steadily decreases from there as it exits the SiO2 and enters the Si. As seen in the graph above for range and straggle vs. energy, increasing energy will result in deeper implants and a shifting of the Gaussian curve to the right, while also flattening it due to increased straggle.

Junction Depth

The junction depth, xj, is the depth at which the impurity concentration equals the substrate concentration, or:

Nimp(xj)=Nsub

This is the location of the p-n junction, where NA=ND. To find this depth, simply find the substrate concentration, trace a horizontal line on the concentration vs. depth plot, find the intersection of the two, then simply draw another line vertically down to the x-axis. This is the junction depth.

Desired junction depth can be achieved by choosing the energy and species based off their range and straggle values:

Nsub=Nimp(xj)=N0exp(−(xj−Rp)22ΔR2p)ln(NsubN0)=−(xj−Rp)22ΔR2p(xj−Rp)2=2ΔR2pln(N0Nsub)xj=Rp±√2ΔR2pln(NsubN0)

Based on this, there can be two junction depths if a more complete Gaussian curve describes the concentration and/or the substrate concentration is high enough ( N_{\text{imp}}(x = 0)">Nsub>Nimp(x=0)).

One rudimentary way of calculating junction depth is the groove-and-stain method. After grinding a shallow cylindrical groove into the substrate surface, a dye is applied that only reacts with one type of present impurity, staining that area. There will then be two distinct areas: one dyed and one plain. The junction depth can then be found using basic geometry:

xj=√R2−w224−√R2−w214

Groove-and-stain method (Source: Texas A&M University ECEN 472 Laboratory Manual)Stopping

Once the ions enter the lattice, they have to stop somehow. Two mechanisms cause stopping: nuclear stopping and electronic stopping. The stopping power is then just the sum of both the nuclear and electron stopping components, measured in energy loss per unit path length of the ion:

S(E)=Sn+Se=dEndx+dEedx

Stopping power as function of ion velocity (ion energy) (source)

More detailed derivations can be found here.

Nuclear Stopping

Nuclear stopping is caused by the collision of the ion into the substrate's nuclei. If the incoming ion's energy exceeds the substrate's displacement energy, the substrate atom will break its four covalent bonds and become displaced from its original position. Depending on the ion's initial energy, a collision cascade can occur: both the ion and first-displaced atom will knock into more atoms and displace those. This continues until both the ion energy and last-displaced atom energy are less than the displacement energy.

Heavier atoms, such as As+, will slow down primarily because of nuclear stopping. Their collisions with substrate atoms are more elastic, so energy loss will be less than the electronic component. For low energies (low velocity), nuclear stopping is the primary mechanism up to a certain point, but electronic stopping will eventually take over.

Nuclear stopping damages the lattice (even causing sputtering) and calls for an activation anneal.

Electronic Stopping

Electron stopping is caused by the interaction between the ion and both bound and free electrons (electron cloud) within the substrate. Given that all implantable ions are cations, they will all react similarly to electrons. The ion continually slows down due to the attraction to/drag of the electrons, eventually stopping.

High velocity ions slow down primarily due to electronic stopping. Their collisions with substrate atoms are highly inelastic. Electronic stopping dominates at moderate energies.

The approximation for electron stopping power is:

Se=4⎛⎝Z7/61Z2(Z2/361+Z2/32)3/2⎞⎠a0Nv

where Z1 is the ion atomic number, Z2 is the substrate atomic number, a0 is the Bohr radius, N is the substrate atomic density (atoms per unit volume), and v is the ion velocity. (I can't seem to find a source on this. Seems to be from one of Lin53, Lin61a, Lin61b from this references section, but even digital copies don't seem to exist.)

Activation Anneal

Because of the damage caused by nuclear stopping, the lattice must be repaired to improve conduction and integrate the impurities into the lattice. The defects that result are primarily vacancies and interstitials. In a vacancy defect, a lattice point is missing an atom: this is caused by either the ion or substrate atom collision (as in the collision cascade). However, this missing atom leaves a place for an implanted ion to become part of the lattice structure, thus "activating" it (the activation energy has to be met). Interstitials refer to atoms that are present in the lattice, but not at a normal lattice point.

Annealing is a high-temperature process, and with high temperature comes movement from the atoms. The previous Gaussian estimation can then be modeled by including the ion diffusivity multiplied by the anneal time in the straggle terms:

N(x)=ϕ√ΔR2p+2Diont√2πexp(−(x−Rp)22(ΔR2p+2Diont))

While skewness remains 0, straggle and kurtosis increase as the curve flattens out.

EquipmentSources

The most common dopants are boron, phosphorus, and arsenic. Aluminum and antimony are occasionally used, but their high diffusivity pose issues for devices.

Gases, such as AsH3, PH3, and BF3, are used to form the ions. Gas is flowed into a small chamber that contains a basic thermionic filament. Electrons are emitted and collide with the gas, ionizing it and forming positive ions within the chamber.

Ionization is the process of adding or removing an electron from a neutral atom, e.g., F becomes F -. In elements that lose an electron, the ionization energy is the amount of energy required to remove one electron from the atom's valence band. To remove the first of He's two valence electrons, 24.6 eV (eV is electronvolts, the amount of energy produced by accelerating one electron through one volt). Ionization energy increases for each valence electron removed. For example, to remove Ne's eighth valence electron requires 239 eV, compared to 21.5 eV for its first. For an implant gas like arsine (AsH3), around 15 eV is required to dissociate the molecule into a As+ cation (positively-charged ion).

Multiple ions can form from the same source gas. For example, BF3 may ionize into 10B+, 11B+, 10BF+, 11BF+, F2+, 10BF2+, and 11BF2+. To find the relative percentage of each in the plasma, the gas spectrum, which shows the beam current vs. atomic masses, needs to be analyzed.

Gas spectrum of BF3 (source)

To increase the probability of ionization, magnets are used to make the electron follow a helical path from one end to the other, rather than just a straight line from anode to cathode.

This continuous ionization of the source gas forms a plasma within the ion source chamber. A vacuum line connected to a vacuum pump is connected to maintain millitorr levels of pressure. Once there is a sufficient amount of ions to form the beam, a large voltage is used to extract the positive ions from the chamber and direct them towards the mass analyzer.

Safety

Most source gases are highly toxic and carcinogenic. See PEL/TLV data for almost all implant gases here.

Mass Analyzer

There are two common types of mass filters: the magnetic sector and radio frequency quadrupole (RFQ). The magnetic sector uses a magnetic field to curve the various species orthogonally from their original path, using the equation:

qvB=mv2r→r=1B√2mVextq

Where B is the magnetic field strength, m is the species' mass, Vext is the extracting voltage, v=2qVextm is the ion velocity, and q is the species' charge. Depending on the species' mass, some may go less than, equal to, or greater than the desired angle.

Magnetic sector operation (source)

While the magnetic sector is a relatively robust method, it does pose issues. Take a P+ cation and a diatomic O2+ cation. P+ has an atomic mass of 30.9738 amu (an electron has a mass of 0.0005 amu, which is negligible). O2+ has an atomic mass of 31.9988 amu. Given a constant extraction voltage and magnetic field strength, the differences in radii will be only 1.61%. This minute difference can cause oxygen to be present in the beam, possibly ruining the implantation.

Acceleration

After the desired ions have been sorted from the undesired, they are accelerated, collimated, and condensed into the final beam.

The acceleration column is kept under as high of vacuum as possible to minimize the amount of stray particles the beam hits, which can cause it to be deflected. The final ion energy is a simple calculation:

Ef=q(Vext+Vacc)

Graphite apertures are commonly used in forming the beam. By designing slits correctly in successive apertures, parts of the beam that stray out are eliminated, leaving only the most parallel ions. They are replaced as needed.

Endstation

The endstation is where the actual implantation takes place. Because each substrate has millions of places (read: transistors) that need implanting and the ion beam is not large enough to cover the entire substrate uniformly, scanning needs to occur. (Sometimes the term rastering is used synonymously, but that is a specific type of scanning. "Scanning" will be used to avoid confusion.)

Beam Stop

Because the magnetic sector only distinguishes based on mass and not charge, some neutrals may be a part of the beam. In order to prevent them from being implanted, some implanters include a small bend at the end of the acceleration column, where an electric field deflects the charged particles one way while the neutrals continue on to nowhere.

Mechanical Scanning

In mechanical scanning, the substrate is attached via clamping or centrifugal force to a robotic arm and moves around a stationary beam. The substrate holder also needs tilting capabilities, otherwise the tilt angle will be 0° (beam is naturally orthogonal to substrate). The primary advantage is a constant tilt angle to ensure consistent implanting across the wafers.

Mechanical scanning is preferred for high dose implants, which use high beam currents.

Electrostatic Scanning

In electrostatic scanning, variable electric fields in both the x and y direction scan the beam across a stationary wafer. This requires a narrow beam, which is generally given by low dose and low current implants. Electrostatic scanning is avoided on high current tools because the beam tends to expand with higher currents, causing poor scanning control. The tilt angle will also vary as the beam scans over the wafer, causing channeling and uniformity issues.

Electrostatic and semi-electrostatic/mechanical scanning (source)

Dose Measurement

To actively measure the dose, a Faraday cup is used. Using the equation from the dose vs. concentration section, the dose is able to be found using an integrator:

ϕ=1qA∫I(t)dt

Plasma Flood Gun

When positive ions strike an insulator, e.g., SiO2, an electric field can form. If this field becomes large enough, the device can be damaged or even ruined altogether. To prevent this electric field from forming, the positive ions need to be neutralized simply by adding an electron. To this, a plasma is struck using the same method as in the ion source: a filament emits electrons that ionize the nearby gas(es). Ar is commonly used. The plasma, being filled with electrons, diffuses into the beam path and the electrons combine with the positive ions directly before they reach the the substrate.

Varian, arguably the leader in ion implantation technology, has published a paper explaining the plasma flood gun and its operation here.

Vacuum

High vacuum is required to prevent beam deflection or contaminants being implanted. This paper discusses vacuum system design for ion implanters.

Process ConsiderationsImplant Masks

Masks are used to block certain areas from being affected by a process. In ion implantation, a mask is used to prevent specific areas of the substrate from being implanted. After all, the source and drain of MOSFET wouldn't necessarily be a source and drain if everywhere around it was doped the same.

So, how to decide which mask to use? A few things about the mask must be considered: cost, stopping power, adhesion to the substrate surface, compatibility with the substrate itself (for example, Cu diffuses very quickly into Si, which can pose later issues in devices). Amorphous (non-crystalline) and high-density materials are best due to larger stopping powers. Common masks include SiO2, Si3N4, photoresist (specific science here), Ti, and W.

Masks can also be used when forming shallow implants. Simply by knowing Rp and ΔRp of the mask and substrate, the energy can be adjusted to push the ions completely past the mask and only a small depth into the substrate. This will take trial-and-error, as the ion energy as they leave the mask is unknown.

Photoresist Masks

Photoresist masks should be chosen based on a few parameters. First, the temperature at which it softens. If being used at a mask, it is likely to have already been post-develop-baked (also called hard bake). Any temperature above the hard bake temperature may cause reflow, which can easily ruin the entire process. There are multiple ways to avoid this:

  • If possible, choose a resist with as high of a softening temperature as possible.
  • Improve cooling to the tool's (where the implanting actually occurs). Instead of using deionized water as a coolant, a mixture of ethylene glycol and water or simply pure ethylene glycol may work better.
  • Choose a metal or dielectric mask that won't have reflow issues. This will add more steps (deposition, etching, photoresist removal) in the fabrication process, but is much better than a completely-ruined process.

High energies can cause strong cross-linking of the resist, making removal more difficult than normal. However, because high-energy ions don't make it to the bottom of the resist, a lift-off process can be performed to more easily remove the strongly-cross-linked parts of the resist.

Lift-off process to remove strongly-cross-linked photoresist

Dielectric and Metal Masks

If the photoresist mask has been ruled out, the other two options are dielectric, such as SiO2 or Si3N4, or metals, such as Ti, W, or silicides of either (TiSi2, WSi2). Range and straggle of the implanted species into the selected material should be chosen. Denser materials are obviously better masks.

Boron ranges in various masking materials (source)

Other materials have been analyzed using a range of energies and species:

Range and straggle of various species into SiO2 (source)

 

Range and straggle of various species into AZ111 photoresist (source)

Using data like this, both the mask thickness and material can be chosen based on the species and energy.

Channeling

Channeling occurs when the ion beam is angled such that there are few obstacles, in this case, atoms, in its path. Take B+. Its atomic radius is around 85 pm ("p" being pico, or 10-12). Viewing the diamond cubic structure of Si from the [110] direction provides very wide openings, or channels, for the ion beam to penetrate. Ions can travel extremely far into the lattice if the beam angle is orthogonal to the axis.

Si lattice as viewed from [110] direction (source)

There are a few methods to prevent channeling:

  • Tilting: By tilting the substrate slightly, the beam is no longer orthogonal to the substrate, causing the ions to collide with the atoms and stop more easily. A common tilt angle is 7°.
  • Surface amorphization: Depositing a thin layer of amorphous material will both slow and scatter the ions once they pass through.
  • Substrate amorphization: Implanting inert species, such as Ar, just below the substrate surface.

The mask height and channel width must be considered when tilting. Basic trigonometry requires a tilt angle no larger than:

θtilt=tan−1(wchannelhmask)

In practice, this limitation is essentially negligible. The mask (generally photoresist or a dielectric like SiO2/Si3N4) is between 100-1000 nm and the channel width to be implanted is 10-2000 nm (no sources on either of these despite combing through Google Scholar, but I've heard these numbers thrown around). These give minimum and maximum angles of:

θtilt, small=tan−1(101000)=0.57∘θtilt, large=tan−1(2000100)=87∘

Again, these numbers are extreme. No manufacturer will be creating 10 nm source/drain widths with 1000 nm mask heights, nor 2000 nm S/D widths with 100 nm mask heights. More appropriate numbers would be 100 nm mask for 10 nm width and 1000 nm mask with 2000 nm width, giving:

θtilt, small=tan−1(10100)=5.71∘θtilt, large=tan−1(20001000)=63.4∘

Additionally, there will be a shadowing effect if any tilt angle is used.

Shadowing as a result of tilting

To fix this, a twist angle is added. The twist rotates the substrates so the shadowed region will be exposed to the ion beam. A "quad" implant may also be performed by implanting at 0, 90°, 180°, and 270°.

Twist and tilt angle (ignore the crystal orientations) (source)

To find the minimum angle (critical angle) to prevent channeling, the following equation is claimed (1, 2, 3):

Ψ=9.73∘√ZionZsubEiond

where Zion/sub are the ion and substrate atomic numbers, E0 is the ion energy in keV, and d is the substrate atomic spacing along the ion path in angstroms. However, the sources that use this equation provide no references to where this came from, nor do the square root argument's dimensions make sense. For this equation to provide units of degrees, the argument needs to be unitless, but it's not:

Ψ=[∘]  ⎷[][][C⋅kg⋅m2s3⋅Cs][m]=[∘] ⎷1[kg⋅m3s2]=[∘]√s2kg⋅m3

This equation has also been claimed for critical angle:

Ψ=√2ZionZsubq2Eiond=√C2⋅s2kg⋅m3

As seen, the dimensions do not cancel out, so neither equation holds in that regard. Dr. Stephen Campbell (author of Fabrication Engineering, one textbook where this equation is used) said via email:

That equation came from the first edition of the book, now more than 20 years ago. As I remember it, it was strictly empirical. You will find a few equations like this in the book. The truth about this field, however inconvenient, is that the development of detailed understanding of fabrication processes often lags a decade or more compared to the development of these processes. I don't know of a theoretical analysis of this topic.

(If you have an original source for this equation, please contact me.)

Uniformity

Modern implanters have both good angle and dose uniformity (see "Representative Ion Implantation Systems" section here): less than 0.1° and one σ < 0.5%. The dose uniformity is primarily a function of the dose measurement system, which can adjust in real-time for any errors that occur (beam turning off, spiking in current, etc).

Doping Alternatives

While doping remains a necessity for device fabrication, ion implantation is not the only method of creating n- or p-type regions.

Diffusion Doping

The other primary method of doping is through diffusion via a furnace. The substrate is heated up and a gas source is introduced into the furnace (chamber or tube). The gases used for each common implant are POCL3 for P, B2H6 for B, and AsH3 and C4H11As (TBA) for As. Using POCL3 as an example, it reacts with O2, to form gaseous P2O5. P2O5 then reacts with the Si surface to form phosphosilicate glass (PSG), (P2O5)x(SiO2)1-x, which serves as the P diffusion source. (The other source gases also react to form a glass layer on Si.)

There are two steps in the diffusion process:

  1. Pre-deposition: The source gas is flowed into the chamber along with a reactive gas (generally O2), forming an oxide and finally the glass layer. Some dopants diffuse slightly into the substrate.
  2. Drive-in: The source gas is turned off and the dopants are allowed to diffuse into the substrate.

Diffusion is based on Fick's laws and increases with temperature:

D(T)=D0exp(−EAkT)

where D0 is the element's pre-factor (temperature-independent).

Spin-On Doping

Spin-on doping (SOD) is the least-common method of doping a substrate. A liquid source is applied to a substrate, then spun at a thousands rpm for a few seconds. This (ideally) provides a uniform film across the surface. The substrate is then heated to allow the dopants to diffuse.

Specific application notes and product information can be found here.

A garage-fab SOD process can be seen here.

See Also

Discuss

Acausal deception

9 мая, 2022 - 00:36
Published on May 8, 2022 9:36 PM GMT

Is the following situation possible under acausal trade? Epistemic status: still trying to understand acausal trade.

Suppose that I am a an agent of "rare type"; it is very unlikely that I exist with the bargaining preferences I have. I conceive that the other agents that I am engaged in acausal trade with are overwhelmingly likely to be of "typical type". I reason (perhaps correctly) that they will believe I am also of typical type. If all agents are of typical type, a particular acausal trade strategy is preferred by all agents. I conceive that if the other agents suspect I am of rare type, they will act in ways that are less optimal for me than if they believe I am of typical type. Luckily, I have reason to believe that they will not suspect I am of rare type, due to the fact I am extremely rare among the range of possible minds: probabilistically speaking, I should not exist. Therefore, I am free to pursue my optimal policy as if I never had to engage in acausal trade and make concessions to ensure that the other agents acted in my favour.



Discuss

Long COVID risk: How to maintain an up to date risk assessment so we can go back to normal life?

8 мая, 2022 - 22:56
Published on May 8, 2022 7:56 PM GMT

Despite Zvi's "Long Long Covid Post" concluding in February that Long COVID risk among healthy, vaccinated individuals is low enough that it's worth pretty much going back to normal life, I haven't felt comfortable doing so given the array of claims to the contrary.

Some of them have surfaced on LessWrong itself:

Others I have come across from friends or on Twitter.

My skills at carefully evaluating scientific research are fairly limited, and I'd also like to avoid spending all of my free time doing so, so I've been kind of stuck in this limbo for now. 

Compounding the challenge of deciding what risks to take is that MicroCOVID doesn't seem to account for the increasing rate of underreporting or the much higher transmissibility of recent Omicron subvariants, making it really hard to decide what level of risk a given activity will pose. And given the transmissibility of those variants, and society's apparent decision to just ... ignore the risk of Long COVID and go back to normal, trying to avoid getting COVID going forward will be more and more socially costly.

I'm sure I'm not the only one in this situation.

So:

  • Is anyone confident going back to normal life despite claims to the contrary without feeling the need to read and evaluate each new study on Long COVID? Why? What logic / heuristics inform that assessment?
    • This seems to be Zvi's current stance, given he seems to be focused elsewhere with his recent posts, so Zvi, if you're reading this, I'd be curious to hear your thoughts!
  • Has anyone been tracking claims to the contrary and assessing their validity (e.g. based on the sorts of critiques Zvi covered in his post)?
  • Would anyone be interested in contributing to a systematic effort to do so? 
    • Could we start some sort of centralized database of studies on Long COVID (a spreadsheet? a wiki?) and folks grab one or two here and there, evaluate them, and note their assessment / rationale?
    • Would folks be interested in contributing to a Kickstarter or something to pay a researcher (e.g. Elizabeth, Zvi, Scott - I don't know if any of them have bandwidth / a price at which they would be interested in doing this currently, but worth asking, or maybe there are other folks with the right skillset/epistemics) to do this?
  • Any other ideas?


Discuss

Demonstrating MWI by interfering human simulations

8 мая, 2022 - 20:28
Published on May 8, 2022 5:28 PM GMT

TLDR: A demonstration that, given some reasonable assumptions, a quantum superposition of a brain creates an exponential number of independent consciousnesses, and each independent consciousness experiences reality as classical.

The demonstration

I'm not going to explain the Many World's Interpretation of Quantum Physics, since much better expositions exist. Feel free to suggest your favourite in the comments.

Imagine we have the ability to simulate a conscious human in the future. This is almost definitely theoretically feasible, but it is unknown how difficult it is in practice.

I take it as obvious that a simulation of a conscious human will be conscious, given that it will answer exactly the same to any questions about consciousness as an actual human. So you should be as convinced it is conscious as you are about any other human.

Now we create a function that takes an input of length n bits and produces as an output whatever a given simulation responds to that input. For example the input could be an index into a gigantic list of philosophy questions, and the output would be whatever Eliezer Yudkowsky would have replied if you had texted him that question at exactly midnight on the 1st of January 2020.

We want to find the exact number of inputs that produce an output with a given property - e.g. the number of inputs that produce a yes answer, or the number where the answer starts with a letter from the first half of the alphabet, or whatever.

Classically it's obvious the only way to do this is to run the function 2^n times - once for every possible input, and then count how many have the desired property. Doing this will require creating 2^n separate consciousness, each of which live for the duration of the function call.

However, on a quantum computer we only need to run the function O(2^n/2) times using a quantum counting algorithm. So for example, if the inputs were 20 bits long, a classical computer would have to call the simulation function roughly a million times whilst the quantum computer would only use on the order of a thousand calls.

The way this works is by creating a superposition of all possible inputs, and then repeatedly applying the function in such a way as to create interference between the outputs which eventually leaves only the desired number.

So for any given property a simulation can exhibit, we can count how many of the 2^n possible simulations have that property in less than 2^n calls. And conceptually there's no possible way to know that without running every single possible simulation.

Now if you agree that anything which acts exactly the same as a conscious being from an input output perspective must in fact be conscious, rather than a philosophical zombie, it seems reasonable to extend that to something which act exactly the same as an aggregate of conscious beings - it must in fact be an aggregate of conscious beings. So even though we've run the simulation function only a thousand times, we must have simulated at least million consciousnesses, or how else could we know that exactly 254,368 of them e.g. output a message which doesn't contain the letter e?

The only way this is possible is if each time we ran the simulation function with a superposition of all possible inputs, it creates a superposition of all possible consciousnesses.

Now each of those consciousnesses produce the same output as they would in a classical universe, so even though they exist in superposition, they themselves see only a single possible input. To them it will appear as though when they looked at the input the superposition collapses, leaving only a single random result, but on the outside view we can see that the whole thing exists in a giant superposition.

This strongly mirrors how the MWI says the entire universe is in a giant superposition, but each individual consciousness sees a collapsed state due to decoherence preventing them interfering with other consciousnesses.



Discuss

Notes from a conversation with Ing. Agr. Adriana Balzarini

8 мая, 2022 - 18:56
Published on May 8, 2022 3:56 PM GMT

This post is part of my "research" series.

These are my notes from a 2022-04-22 conversation (in Spanish) with Adriana Balzarini Ingeniera Agrónoma. I met Adriana as part of the Board of Directors of the Partido de la Costa Science Club. The conversation wasn't recorded, so I've had to reconstruct some of the arguments from memory, and taken the opportunity to reformat and abridge the content. I thank her for the generous donation of so much of her time.

The Science Club ceased activities back in 2015 due to inhability to form a Board of Directors. We've managed to keep organizing voluntary participation in scientific work under the municipal government. The work we do is very fieldwork-oriented, so I don't know how much I can help you with bibliographic research.

The way we operate is as follows: our working group interfaces with research institutions such as CONICET (Consejo Nacional de Investigaciones Científicas y Técnicas, National Scientific and Technical Research Council) on behalf of the municipal government, and we offer our services for studies that need local data but would normally find it expensive -and early during the COVID-19 pandemic, outright impossible- to obtain. We then meet with the study authors to discuss goals and methodology, which then informs the fieldwork of our volunteers. Our work usually ends up credited as collaboration. The Club used to have some public presence, with school talks and a blog, but the restructuring depreoritized outreach activities.

As for the actual question of research, I rely heavily on expert opinion. I start by keyword-searching on Google/SciELO[1], which nets me relevant papers, organizations (specially research institutions), and -critically- researcher contact information. I then get in touch through whichever means they have made public and ask them for details of my topic of interest. Obviously I do some reading first; showing up for an interview knowing absolutely nothing is disrespectful and a waste of the expert's time.

It may be a different experience on mathematics or the humanities, but I doubt it. Knowledge is a collaborative enterprise. If you want to participate, you need to look for the communities where that knowledge lives. Even in dead disciplines you can find people studying them from a historical perspective. You'll notice you're there when you meet people who speak the jargon. Words that you and I use everyday -such as "error"- will take on special, well-defined meanings; this is the mark of a community that's trying to make knowledge transmissible.

I haven't had much of a working relationship with the local libraries. They aren't really research libraries, they are oriented towards a wider community role. If they even have a subject catalog, I'm not familiar with it. They probably have a digital catalog of some kind, since the government has been pushing their Aguapey library management system on every public library, but I think the libraries aren't obligated to make that publicly accessible.

  1. For a description of the SciELO online library, see
    Canales et al. SciELO: A cooperative project for the dissemination of science. 2009. Revista Española de Sanidad Penitenciaria. ISSN: 2013-6463. Available at SciELO. ↩︎



Discuss

Страницы