Новости LessWrong.com

A community blog devoted to refining the art of rationality
Обновлено: 21 час 10 минут назад

Open Philanthropy is seeking proposals for outreach projects

17 июля, 2021 - 00:19
Published on July 16, 2021 9:19 PM GMT

[Cross-posted from the EA Forum.]

Open Philanthropy is seeking proposals from applicants interested in growing the community of people motivated to improve the long-term future via the kinds of projects described below.[1]

Apply to start a new project here; express interest in helping with a project here.

We hope to draw highly capable people to this work by supporting ambitious, scalable outreach projects that run for many years. We think a world where effective altruism, longtermism, and related ideas are routine parts of conversation in intellectual spaces is within reach, and we’re excited to support projects that work towards that world.

In this post, we describe the kinds of projects we’re interested in funding, explain why we think they could be very impactful, and give some more detail on our application process.

Proposals we are interested in Programs that engage with promising young people

We are seeking proposals for programs that engage with young people who seem particularly promising in terms of their ability to improve the long-term future (and may have interest in doing so).

Here, by “particularly promising”, we mean young people who seem well-suited to building aptitudes that have high potential for improving the long-term future. Examples from the linked post include aptitudes for conducting research, advancing into top institutional roles, founding or supporting organizations, communicating ideas, and building communities of people with similar interests and goals, among others. Downstream, we hope these individuals will be fits for what we believe to be priority paths for improving the long-term future, such as AI alignment research, technical and policy work reducing risks from advances in synthetic biology, career paths involving senior roles in the national security community, and roles writing and speaking about relevant ideas, among others.

We’re interested in supporting a wide range of possible programs, including summer or winter camps, scholarship or fellowship programs, seminars, conferences, workshops, and retreats. We think programs with the following characteristics are most likely to be highly impactful:

• They engage people ages 15 - 25 who seem particularly promising in terms of their ability to improve the long-term future, for example people who are unusually gifted in STEM, economics, philosophy, writing, speaking, or debate.
• They cover effective altruism (EA), rationality, longtermism, global catastrophic risks, or related topics.
• They involve having interested young people interact with people currently working to improve the long-term future.

Examples of such programs that Open Philanthropy has supported include SPARC, ESPR, the SERI and FHI summer research programs, and the recent EA Debate Championship. However, we think there is room for many more such programs.

We especially encourage program ideas which:

• Have the potential to engage a large number of people (hundreds to tens of thousands) per year, though we think starting out with smaller groups can be a good way to gain experience with this kind of work.
• Engage with groups of people who don’t have many ways to enter relevant intellectual communities (e.g. they are not in areas with high concentrations of people motivated to improve the long-term future).
• Include staff who have experience working with members of the groups they hope to engage with—in particular, experience talking with young people about new ideas while being respectful of their intellectual autonomy and encouraging independent intellectual development.

We encourage people to have a low bar for submitting proposals to our program, but note that we view this as a sensitive area: we think programs like these have the potential to do harm by putting young people in environments where they could have negative experiences. Nicole Ross at the Centre for Effective Altruism (email nicole@centreforeffectivealtruism.org) is available to provide advice on these kinds of risks.

Some reasons why we think this work has high expected value

A priori, we would guess that people are more likely to get interested in new ideas and opportunities when they are relatively young and have fewer preexisting commitments. This guess is consistent with the results of a survey Open Philanthropy recently ran—we surveyed approximately 200 people who our advisors suggested had the potential to do good longtermist work, most of whom had recently made career changes that we thought were positive from a longtermist perspective. As part of this survey, we asked respondents several questions regarding the age at which they first encountered effective altruism or effective altruism-adjacent ideas.

• On average, survey respondents reported first encountering EA/EA-adjacent ideas when they were 20 years of age.
• About 25% of respondents first encountered EA/EA-adjacent ideas at ages 18 or below, even though few EA outreach projects focus on that age range.
• On average, respondents said the best age for them to first encounter EA/EA-adjacent ideas would have been 16.

Survey respondents often mentioned that hearing about EA before starting university would have been particularly helpful because they could have planned how to use their time at university better, e.g. what to major in.

We also asked survey respondents to brainstorm open-endedly about how to get people similar to them interested in these ideas. 10% of responses mentioned starting outreach programs younger, particularly in high school. Several respondents mentioned that SPARC and ESPR had been helpful for them and that they would recommend these programs to similar people. (Certain other high school outreach projects have reported less success, but we don’t think these less-targeted programs provide much evidence about how promising targeted high school outreach is likely to be overall, as discussed here.)

Our survey also showed that EA groups, particularly university groups, have had a lot of impact on longtermist career trajectories. On a free-form question asking respondents to list the top few things that increased their expected impact, respondents listed EA groups more commonly than any other factor. On other measures of impact we used in our survey analysis, EA groups came between second and fourth in potential factors, above many EA organizations and popular pieces of writing in the EA-sphere. Most of this impact (65 - 75% on one measure) came from university groups. We think this suggests that, more generally, offering high-quality opportunities for university students to get involved is a promising kind of intervention.

Made-up examples of programs we think could be impactful

These examples are intended to be illustrative of the kinds of programs we’d be interested in funding. This is not intended to be a comprehensive list, nor a list of the programs we think would be most impactful.

We think these programs are unlikely to work fully as written. Founders generally have to dive deep into a project plan to figure out what’s tenable, altering their plan multiple times as they get a better understanding of the space, and we haven’t done that work. As such, we’d like these examples to serve as inspiration, not as instructions. We think programs of this kind are more likely to be successful when the founders develop their own vision and understanding of their target audience.

We would ultimately like to support dedicated teams or organizations that run programs for young people at scale. That said, we are likely to recommend that applicants with less of a track record start by trying out a small pilot of their program and iterating while maximizing program quality and target fit, rather than scaling immediately.

Example 1: A free two-week summer school in Oxford that teaches content related to longtermism to promising high school students. The program could have a similar structure to SPARC and ESPR, but with a more explicitly longtermist focus, and it could engage a broader range of gifted high school students.

• We think programs like this are most effective when they focus on highly promising students, e.g. by filtering on Olympiad participation, high standardized test scores, competitive awards, or other markers of talent.
• Oxford seems like a good location for programs like this because its status as an EA hub makes it easy for current longtermists doing good work to instruct and interact with students, which we think is important for programs like this to be successful. (Berkeley and Stanford seem like good locations for similar reasons.)
• Oxford is a cool place to visit in and of itself, making a program located there attractive as a paid trip for high school students.

Example 2: A monthly AI safety workshop for computer science undergraduates, covering existing foundational work in AI safety.

• There have been several programs like this, notably AIRCS, which our survey suggests has had an impact on some longtermist career trajectories. We think it’s likely that AIRCS hasn’t saturated the pool of top computer science undergraduates, and that there is room for more programs of this form that experiment with different kinds of content and instructors.

Example 3: A one-week summer program about effective altruism in Berkeley combined with a prestigious 20,000 merit-based scholarship for undergraduate students. The scholarship would involve an application process that required substantial engagement with ideas related to effective altruism, e.g. a relevant essay and an interview. • We think the best scholarship programs will be fairly selective, so as to attract very promising applicants and create a very strong cohort. • In-person programs that run right before students start their undergraduate degrees might be particularly impactful, via bolstering EA groups at top universities. • Scholarships and other programs that include substantial financial opportunities risk attracting applicants that are only interested in the money provided by the program. We think programs like this should construct application processes that make an effort to identify applicants genuinely interested in effective altruism, e.g. via essays and interviews. Example 4: A monthly four-day workshop teaching foundational rationality content to promising young people. • The workshop could teach foundational technical topics in rationality, including some covered by CFAR in the past, e.g. probability theory, Bayesianism, Fermi estimation, calibration, betting, cognitive biases, etc., as well as exercises intended to help students use these thinking tools in the real world. • This could overlap heavily with SPARC’s content, but could engage a larger number of people per year than SPARC has capacity for, as well as a more varied or substantively different audience. Example 5: A fall jobs talk and follow-up discussion that’s held at top universities describing career paths in defensive work for future biological catastrophes. • We think fall final year is a good time to prompt undergraduate students with concrete career suggestions, and with COVID-19 in recent memory, we think the next few years could be a particularly good time to talk to students about careers in global catastrophic biological risk reduction. Projects aiming at widespread dissemination of relevant high-quality content We are also seeking proposals for projects that aim to share high-quality, nuanced content related to improving the long-term future with large numbers of people. Projects could cover wide areas such as effective altruism, rationality, longtermism, or global catastrophic risk reduction, or they could have a more specific focus. We’re interested in supporting people both to create original content and to find new ways to share existing content. Potential project types include: • Podcasts • YouTube channels • Massive open online courses (MOOCs) • New magazines, webzines, blogs, and media verticals • Books, including fiction • Strategic promotion of existing content (with the permission of the creators of the content, or their representatives), especially those that have historically drawn in promising individuals Existing projects along these lines include the 80,000 Hours Podcast, Robert Miles’s AI alignment YouTube channel, and Vox’s Future Perfect. We encourage projects that involve content in major world languages other than English, especially by native speakers of those languages—we think projects in other languages are especially likely to reach people who haven’t had as many opportunities to engage with these ideas. We would like interested people to have a low bar for submitting a proposal, but we think projects that misrepresent relevant ideas or present them uncarefully can do harm by alienating individuals who would have been sympathetic to them otherwise. We also think it’s important to be cognizant of potential political and social risks that come with content creation and dissemination projects in different countries. Nicole Ross at the Centre for Effective Altruism (email nicole@centreforeffectivealtruism.org) is available to provide advice on these kinds of risks. Some reasons why we think this work has high expected value Our sense from talking to people doing longtermist work we think is promising has been that, for many, particular pieces of writing or videos were central to their turn towards their current paths. This seems broadly in line with the results of the survey we conducted mentioned above. The bodies of written work of Nick Bostrom, Eliezer Yudkowsky, and Peter Singer were in the top 10 sources of impact on longtermist career trajectories (of e.g. organizations, people, and bodies of work) across several different measures. On one measure, Nick Bostrom’s work by itself had 68% of the impact of the most impactful organization and 75% of the impact of the second most impactful organization. When asked what outreach would attract similar people to longtermist work, 8% of respondents in the survey gave free-form responses implying that they think simply exposing similar people to EA/EA-adjacent ideas would be sufficient. These data points suggest to us that even absent additional outreach programs, sharing these ideas more broadly could ultimately result in people turning towards career activities that are high-value from a longtermist perspective. For many who could work on idea dissemination, we think increasing the reach of existing works with a strong track record, like those given above, may be more impactful per unit of effort than creating new content. Made-up examples of projects we think could be impactful As above, these examples are intended to be illustrative of the kinds of programs we’d be interested in funding. This is not intended to be a comprehensive list, nor a list of the programs we think would be most impactful. We think these programs are unlikely to work fully as written and would like these projects to serve as inspiration, not as instructions. Example 1: Collaborations with high-profile YouTube creators to create videos covering longtermist topics. • We think YouTube is an attractive promotional platform because different creators come with different audiences, making it easy to share with the kinds of people who most often become interested in longtermist ideas. Example 2: Targeted social media advertising of episodes of the 80,000 Hours podcast. The project would aim to maximize downloads of the 80,000 Hours Podcast episodes that go through social media referrals. • The 80,000 Hours podcast seems promising to promote because we think it’s high-quality, quick to consume, and varied enough in content to appeal to a fairly wide audience. • The project could experiment with indiscriminately advertising podcast episodes to promising early-career individuals, e.g. STEM, economics, or philosophy students, or with advertising select podcast episodes on particular topics to audiences that they may appeal to. • Any project of this form should be done in collaboration with 80,000 Hours. Example 3: A website that delivers free copies of physical books, e-books, or audiobooks that seem helpful for understanding how to do an outsized amount of good to people with a .edu email address who request them. • The bulk of the project work could be focused on website design and advertising, while book distribution could be handled through EA Books Direct, or done as part of this project. Example 4: A MOOC covering existing AI safety work. Example 5: A new magazine that covers potentially transformative technologies and ways in which they could radically transform civilization in positive or negative ways. Application process Primary application If you think you might want to implement either of the kinds of outreach projects listed above, please submit a brief pre-proposal here. If we are interested in supporting your project, we will reach out to you and invite you to submit more information. We encourage submissions from people who are uncertain if they want to found a new project and just want funding to seriously explore an idea. If it would be useful for applicants developing their proposals, we are open to funding them to do full-time project development work for 3 months. We are happy to look at multiple pre-proposals from applicants who have several different project ideas. We may also be able to help some applicants (e.g. by introducing them to potential collaborators, giving them feedback about plans and strategy, providing legal assistance, etc.) or be able to help find others who can. We are open to and encourage highly ambitious proposals for projects that would require annual budgets of millions of dollars, including proposals to scale existing projects that are still relatively small. We intend to reply to all applications within two months. We have also been in touch with the Effective Altruism Infrastructure Fund and the Long-Term Future Fund, and they have expressed interest in funding proposals in the areas we describe below. If you want, you can choose to have them also receive your application via the same form we are using. There is no deadline to apply; rather, we will leave this form open indefinitely until we decide that this program isn’t worth running, or that we’ve funded enough work in this space. If that happens, we will update this post noting that we plan to close the form at least a month ahead of time. Collaborator application If you aren’t interested in starting something yourself, but you would be interested in collaborating on or helping with the kinds of outreach projects listed above (either full or part-time), let us know here. We will connect you to project leads if we feel like there is a good fit for your skills and interests. If you have any questions, please contact longtermfuture-outreach-rfp@openphilanthropy.org. 1. Our work in this space is motivated by a desire to increase the pool of talent available for longtermist work. We think projects like the ones we describe may also be useful for effective altruism outreach aimed at other cause areas, but we (the team running this particular program, not Open Philanthropy as a whole) haven’t thought through how valuable this work looks from non-longtermist perspectives and don’t intend to make that a focus. ↩︎ Discuss [AN #156]: The scaling hypothesis: a plan for building AGI 16 июля, 2021 - 20:10 Published on July 16, 2021 5:10 PM GMT [AN #156]: The scaling hypothesis: a plan for building AGI Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world View this email in your browser Newsletter #156 Alignment Newsletter is a weekly publication with recent content relevant to AI alignment around the world. Find all Alignment Newsletter resources here. In particular, you can look through this spreadsheet of all summaries that have ever been in the newsletter. Audio version here (may not be up yet). Please note that while I work at DeepMind, this newsletter represents my personal views and not those of my employer. SECTIONS ﻿HIGHLIGHTS ﻿TECHNICAL AI ALIGNMENT ﻿AGENT FOUNDATIONS ﻿AI GOVERNANCE ﻿ ﻿ ﻿ HIGHLIGHTS The Scaling Hypothesis (Gwern Branwen) (summarized by Rohin): This post centers around the scaling hypothesis: Once we find a scalable architecture which can be applied fairly uniformly, we can simply train ever larger networks and ever more sophisticated behavior will emerge naturally as the easiest way to optimize for all the tasks and data. More powerful NNs are “just” scaled-up weak NNs, in much the same way that human brains look much like scaled-up primate brains⁠. Importantly, we can get this sophisticated behavior just by training on simple objectives, such as “predict the next word”, as long as the data is sufficiently diverse. So, a priori, why might we expect the scaling hypothesis to be true? The core reason is that optimal (or human-level) prediction of text really does require knowledge, reasoning, causality, etc. If you don’t know how to perform addition, you are probably not going to be able to predict the next word in the sentence “Though he started with six eggs, he found another fourteen, bringing his total to ____”. However, since any specific fact is only useful in a tiny, tiny number of cases, it only reduces the expected loss by a tiny amount. So, you’ll only see models learn this sort of behavior once they have exhausted all the other “easy wins” for predicting text; this will only happen when the models and dataset are huge. Consider a model tasked with predicting characters in text with a set of 64 characters (52 uppercase and lowercase letters, along with some punctuation). Initially it outputs random characters, assigning a probability of 1/64 to the correct character, resulting in a loss of 6 bits. Once you start training, the easiest win is to simply notice how frequent each character is; just noticing that uppercase letters are rare, spaces are common, vowels are common, etc. could get your error down to 4-5 bits. After this, it might start to learn what words actually exist; this might take 10^5 - 10^6 samples since each word is relatively rare and there are thousands of words to learn, but this is a drop in the bucket given our huge dataset. After this step, it may have also learned punctuation along the way, and might now be down to 3-4 bits. At this point, if you sample from the model, you might get correctly spelled English words, but they won’t make any sense. With further training the model now has to pick up on associations between adjacent words to make progress. Now it needs to look at things 10 characters ago to predict the next character -- a far cry from our initial letter frequencies where it didn’t even need to look at other characters! For example, it might learn that “George W” tends to be followed by “ashington”. It starts to learn grammar, being able to correctly put verbs in relation to subjects and objects (that are themselves nouns). It starts to notice patterns in how words like “before” and “after” are used; these can then be used to better predict words in the future; at this point it’s clear that the model is starting to learn semantics. Now the loss is around 2 bits per character. A little more training and your model starts to produce sentences that sound human-like in isolation, but don’t fit together: a model might start a story about a very much alive protagonist and then talk about how she is dead in the next sentence. Training is now about fixing errors like these and each such fix gains a tiny amount of accuracy -- think ten thousandths of a bit. Every further 0.1 bits you gain represents the model learning a huge amount of relevant knowledge (and correspondingly each subsequent 0.1 bits takes a much larger amount of training and data). The final few fractions of a bit are the most important and comprise most of what we call “intelligence”. (The human baseline is a loss of 0.7 bits, with lots of uncertainty on that figure.) So far this is a clever argument, but doesn’t really establish that this will work in practice -- for example, maybe your model has to have 10^100 parameters to learn all of this, or maybe existing models and algorithms are not sophisticated enough to find the right parameters (and instead just plateau at, say, 2 bits of loss). But recent evidence provides strong support for the scaling hypothesis: 1. The scaling laws (AN #87) line of work demonstrated that models could be expected to reach the interesting realm of loss at amounts of compute, data, and model capacity that seemed feasible in the near future. 2. Various projects have trained large models and demonstrated that this allows them to solve tasks that they weren’t explicitly trained for, often in a more human-like way and with better performance than a more supervised approach. Examples include GPT-3 (AN #102), Image GPT, BigGAN, AlphaStar (AN #73), etc. (The full post has something like 25 examples.) The author then argues that it seems like most researchers seem to be completely ignoring this phenomenon. OpenAI is the only actor that really has the conviction needed to put a large amount of resources behind a project based on the scaling hypothesis (such as GPT-3); DeepMind seems to believe in a weaker version where we need to build a bunch of “modules” similar to those in the human brain, but that those modules can then be scaled up indefinitely. Other actors seem to not take either scaling hypothesis very seriously. ﻿ Rohin's opinion: In my view, the scaling hypothesis is easily the most important hypothesis relevant to AI forecasting and AI development models, and this is the best public writeup of it that I know of. (For example, it seems to be an implicit assumption in the bio anchors framework (AN #121).) I broadly agree with the author that it’s a bit shocking how few people seem to be taking it seriously after OpenAI Five, AlphaStar, GPT-3, Copilot, etc. I think this includes the AI safety space, where as far as I can tell the primary effect has been that it is even more fashionable to have shorter timelines, whereas it hasn’t affected AI safety research very much. However, I do know around 3-4 researchers who changed what they were working on based on changing their mind about the scaling hypothesis, so it’s possible there are several others I don’t know about. As a simple example of how the scaling hypothesis affects AI safety research, it suggests that the training objective (“predict the next word”) is relatively unimportant in determining properties of the trained agent; in contrast, the dataset is much more important. This suggests that analyses based on the “reward function used to train the agent” are probably not going to be very predictive of the systems we actually build. ﻿ ﻿ ﻿ TECHNICAL AI ALIGNMENT ﻿ AGENT FOUNDATIONS The Accumulation of Knowledge (Alex Flint) (summarized by Rohin): Probability theory can tell us about how we ought to build agents that have knowledge (start with a prior and perform Bayesian updates as evidence comes in). However, this is not the only way to create knowledge: for example, humans are not ideal Bayesian reasoners. As part of our quest to describe existing agents (AN #66), could we have a theory of knowledge that specifies when a particular physical region within a closed system is “creating knowledge”? We want a theory that works in the Game of Life (AN #151) as well as the real world. This sequence investigates this question from the perspective of defining the accumulation of knowledge as increasing correspondence between a map and the territory, and concludes that such definitions are not tenable. In particular, it considers four possibilities and demonstrates counterexamples to all of them: 1. Direct map-territory resemblance: Here, we say that knowledge accumulates in a physical region of space (the “map”) if that region of space looks more like the full system (the “territory”) over time. Problem: This definition fails to account for cases of knowledge where the map is represented in a very different way that doesn’t resemble the territory, such as when a map is represented by a sequence of zeros and ones in a computer. 2. Map-territory mutual information: Instead of looking at direct resemblance, we can ask whether there is increasing mutual information between the supposed map and the territory it is meant to represent. Problem: In the real world, nearly every region of space will have high mutual information with the rest of the world. For example, by this definition, a rock accumulates lots of knowledge as photons incident on its face affect the properties of specific electrons in the rock giving it lots of information. 3. Mutual information of an abstraction layer: An abstraction layer is a grouping of low-level configurations into high-level configurations such that transitions between high-level configurations are predictable without knowing the low-level configurations. For example, the zeros and ones in a computer are the high-level configurations of a digital abstraction layer over low-level physics. Knowledge accumulates in a region of space if that space has a digital abstraction layer, and the high-level configurations of the map have increasing mutual information with the low-level configurations of the territory. Problem: A video camera that constantly records would accumulate much more knowledge by this definition than a human, even though the human is much more able to construct models and act on them. 4. Precipitation of action: The problem with our previous definitions is that they don’t require the knowledge to be useful. So perhaps we can instead say that knowledge is accumulating when it is being used to take action. To make this mechanistic, we say that knowledge accumulates when an entity’s actions become more fine-tuned to a specific environment configuration over time. (Intuitively, they learned more about the environment and so could condition their actions on that knowledge, which they previously could not do.) Problem: This definition requires the knowledge to actually be used to count as knowledge. However, if someone makes a map of a coastline, but that map is never used (perhaps it is quickly destroyed), it seems wrong to say that during the map-making process knowledge was not accumulating. ﻿ ﻿ ﻿ AI GOVERNANCE AI Certification: Advancing Ethical Practice by Reducing Information Asymmetries (Peter Cihon et al) (summarized by Rohin): Certification is a method of reducing information asymmetries: it presents credible information about a product to an audience that they couldn’t have easily gotten otherwise. With AI systems, certification could be used to credibly share information between AI actors, which could promote trust amongst competitors, or to share safety measures to prevent a race to the bottom on safety, caused by worrying that “the other guys would be even more unsafe”. Certification is at its best when there is demand from an audience to see such certificates; public education about the need for credible information can help generate such demand. However, certification often runs into problems. Symbol-substance decoupling happens when certificates are issued to systems that don’t meet the standards for certification. For example, in “ethics washing”, companies advertise a self-certificate in which their products are approved by ethics boards, but those ethics boards have no real power. Means-ends decoupling happens when the standards for certification don’t advance the goals for which the certificate was designed. For example, a certificate might focus on whether a system was tested, rather than on what test was conducted, leading applicants to use easy-to-pass tests that don’t actually provide a check on whether the method is safe. Effective certification for future AI systems needs to be responsive to changes in AI technology. This can be achieved in a few ways: first, we can try to test the underlying goals which are more likely to remain stable; for example, we could certify ethical principles that will likely remain the same in the future. Second, we can match the certification to the types of people and institutions, that is, our certifications talk about the executives, citizens, or corporations (rather than e.g. specific algorithms, that may be replaced in the future). Third, the certification system can build in mechanisms for updating the certification criteria periodically. The paper then analyzes seven existing certification systems for AI systems; you’ll have to read the paper for details. Case studies of self-governance to reduce technology risk (Jia) (summarized by Rohin): Should we expect AI companies to reduce risk through self-governance? This post investigates six historical cases, of which the two most successful were the Asilomar conference on recombinant DNA and the actions of Leo Szilard and other physicists in 1939 (around the development of the atomic bomb). It is hard to make any confident conclusions, but the author identifies the following five factors that make self-governance more likely: 1. The risks are salient. 2. If self-governance doesn’t happen, then the government will step in with regulation (which is expected to be poorly designed). 3. The field is small, so that coordination is easier. 4. There is support from gatekeepers (e.g. academic journals). 5. There is support from credentialed scientists. Corporate Governance of Artificial Intelligence in the Public Interest (Peter Cihon et al) (summarized by Rohin): This paper is a broad overview of corporate governance of AI, where by corporate governance we mean “anything that affects how AI is governed within corporations” (a much broader category than the governance that is done by corporations about AI). The authors identify nine primary groups of actors that can influence corporate governance and give many examples of how such actors have affected AI governance in the past. The nine groups are managers, workers, investors, corporate partners and competitors, industry consortia, nonprofit organizations, the public, the media, and governments. Since the paper is primarily a large set of examples along with pointers to other literature on the topic, I’m not going to summarize it in more detail here, though I did find many of the examples interesting (and would dive into them further if time was not so scarce). FEEDBACK I'm always happy to hear feedback; you can send it to me, Rohin Shah, by replying to this email. PODCAST An audio podcast version of the Alignment Newsletter is available. This podcast is an audio version of the newsletter, recorded by Robert Miles. Subscribe here: Copyright © 2021 Alignment Newsletter, All rights reserved. Want to change how you receive these emails? You can update your preferences or unsubscribe from this list. Discuss Underlying model of an imperfect morphism 16 июля, 2021 - 16:13 Published on July 16, 2021 1:13 PM GMT .mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} We've already seen that if M0=(F0,Q0) and M1=(F1,Q1) are generalised models, with the relation r⊂W0×W1 a Q-preserving morphism between them, then there is an underlying model Mr=(F0⊔F1,Qr) between them. Since r⊂W0×W1, Qr is defined on r; indeed, it is non-zero on r only. The underlying model has functions r0 and r1 to M0 and M1, which push forward Qr in a unique way - to Q0 and Q1 respectively. Essentially: • There is an underlying reality Mr of which M0 and M1 are different, consistent, facets. Illustrated, for gas laws: Underlying model of imperfect morphisms But we've seen that relations r need not be Q-preserving; there are weaker conditions that also form categories. Indeed, even in the toy example above, the ideal gas laws and the "atoms bouncing around" model don't have a Q-preserving morphism between them. The atoms bouncing model is more accurate, and the idea gas laws are just an approximation of these (for example, they ignore molar mass). Let's make the much weaker assumption that r is Q-birelational - essentially that if any wi has non-zero Qi-measure (i.e. 0">Qi(wi)>0), then r relates it to at least one other wj which also has non-zero Qj-measure. Equivalently, if we ignore all elements with zero Qi-measure, then r and r−1 are surjective relations between what's left. Then we have a more general underlying morphism result: Statement of the theorem Let r be a Q-birelational morphism between M0=(F0,Q0) and M1=(F1,Q1), and pick any 0≤α≤1. Then there exists a generalised model Mαr=(F0⊔F1,Qαr), with Qαr=0 off of r⊂W0×W1 (this Qαr is not necessarily uniquely defined). This has natural functional morphisms r0:Mαr→M0 and r1:Mαr→M1. Those ri push forward Qαr to Mi, such that for the distance metric L defined on morphisms, 1. |r0(Qαr)−Q0|1=αL(r), 2. |r1(Qαr)−Q1|1=(1−α)L(r). By the definition of L, this is the minimum |r0(Qαr)−Q0|1+|r1(Qαr)−Q1|1 we can get. The proof is in this footnote[1]. Accuracy of models If α=0, we're saying that M0 is a correct model, and that M1 is an approximation. Then the underlying model reflects this, with M0 a true facet of the underlying model, and M1 the closest-to-accurate facet that's possible given the connection with M0. If α=1, then it's reversed: M0 is an approximation, and M1 a correct model. For α between those two value, we see both M0 and M1 as approximations of the underlying reality Mr. Measuring ontology change This approach means that L(r) can be used to measure the extent of an ontology crisis. Assume M0 is a the initial ontology, and M1 is the new ontology. Then M1 might include entirely new situations, or at least unusual ones that were not normally thought about. The r connects the old ontology with the new one: it details the crisis. In an ontology crisis, there are several elements: 1. A completely different way of seeing the world. 2. The new and old ways result in similar predictions in standard situations. 3. The new way results in very different predictions in unusual situations. 4. The two ontologies give different probabilities to unusual situations. The measure L amalgamates points 2., 3., and 4. above, giving an idea of the severity of the ontology crisis in practice. A low L(r) might be because because the new and old ways have very similar predictions, or because the situations where they differ might be unlikely. For point 1, the "completely different way of seeing the world", this is about how features change and relate. The L(r) is indifferent to that, but we might measure this indirectly. We can already use a generalisation of mutual information to measure the relation between the distribution Q and the features F. We could use that to measure the relation between F0⊔F1, the features of M1r, and Q1r, its probability distribution. Since Q1r is more strongly determined by Q1, this could[2] measure how hard it is to express Q0 in terms of F1. 1. Because r is bi-relational, there is a Q′1 such that r is a Q-preserving morphism between M0 and M′1(F1,Q′1); and furthermore |Q′0−Q0|1=L(r). Let M0r be an underlying model of this morphism. Similarly, there is a Q′0 such that r is a Q-preserving morphism between M′0=(F0,Q′0) and M1; and furthermore |Q′1−Q1|1=L(r). Let M1r be an underlying model of this morphism. Note that M0r and M1r differ only in their Q0r and Q1r; they have same feature sets and same worlds. Then define Mαr as having Qαr=(1−α)Q0r+αQ1r. Then r0(Qαr)=(1−α)Q0+αQ′0, so |r0(Qαr)−Q0|1=|αQ0−αQ′0|1=α|Q0−Q′0|=αL(r). Similarly, |r1(Qαr)−Q1|1=(1−α)L(r). ↩︎ 2. This is a suggestion; there may be more direct ways of measuring this distance or complexity. ↩︎ Discuss When beliefs become identities, truth-seeking becomes hard 16 июля, 2021 - 12:02 Published on July 16, 2021 9:02 AM GMT This is Rational Animation's script for the video linked above. This time, the topic is the "Rethinking Identity" section in the book "The Scout Mindset" by Julia Galef. As always, feedback is welcome. Previously, we made a video about how scorn for ideas and communities is often damaging for truth-seeking and truth-telling. In this video, we’ll talk about another way in which social reality might interfere with your ability to pursue what's true and what's best for you to do: identity. Julia Galef talks about this topic in her book “The Scout Mindset.” The scout mindset is the mindset of truth-seeking, and she opposes it to the soldier mindset. Having a soldier mindset means defending your beliefs for reasons other than truth as if you were fighting a war of ideas instead of trying to figure out what’s going on. She makes the case that the scout mindset is way more helpful than the soldier mindset. The soldier mindset has advantages, such as avoiding unpleasant emotions, increasing motivation, and persuasion. But Julia Galef outlines strategies to keep the advantages of the soldier mindset without actually sacrificing truth-seeking and truth-telling. Seems cool, right? In her book, there are also useful sections about how to be a better scout. One of those sections is called “rethinking identity”, which is the topic of this video. Julia observes that it’s often the case that certain kinds of beliefs are prone to become identities. Communities form around them and, in a sense, fight to defend them. She makes the example of the “mommy wars”, in which one faction believes that it's better to feed babies with breast milk and the other thinks that baby formula is fine. In theory, this should be a straightforward empirical question, but you can observe the two factions arguing passionately and calling each other epithets in an ideological battle that spans years and transcends geography. There are many other examples you could come up with [on screen]. When your beliefs are central to your identity, it is way more difficult to change your mind if they turn out to be wrong. And the discussions you have will not be about the facts that make those beliefs true or false, but about either how much you identify with them or how your opponent is stupid. For example, you may wear t-shirts with slogans or engage in personal attacks. When someone makes an argument against beliefs that are central to your identity, you may feel attacked even when no personal attack has been made. If you merely disagreed with the person making that argument instead of also identifying with your belief, you wouldn’t feel attacked. Julia Galef observes two ways in which beliefs become identities: when the belief causes the person to feel embattled or proud. Feeling embattled happens when you are part of a group that holds a belief that attracts mockery, scorn, persecution, or stigmatization from the rest of society or from another group. When society holds a negative attitude towards groups that hold a minority belief, the belief becomes even stronger and gets defended more passionately. People in the embattled group will want to stand up for their beliefs even more and will become more cohesive in solidarity towards each other. An obvious example is groups that hold beliefs against the scientific consensus, such as anti-vaccinists and flat-earthers. Hating them passionately and publicly is in many cases going to reinforce their beliefs. If someone writes an article that explains the benefit of vaccines but at the same time heavily mocks anti-vaccinists, who is the audience for that article? Not anti-vaccinists, for sure. This brings us to the next reason why beliefs become identities: feeling proud. The writer of that article against anti-vaccinists is not really talking with them, but he is writing to reinforce his own identity and appeal to his group. He feels proud of his anti-anti-vaccinists identity, preventing him from productively engaging with the other side. Sometimes feeling proud doesn’t come from holding a belief that society considers virtuous but can also be a direct consequence of feeling embattled. In the words of Julia Galef: “When the world around you is constantly shouting about how wrong you are, declaring your superiority can feel like a legitimate response, the only way to oppose such negative messaging.” In her book, Galef proposes a solution to the problem of identity interfering with truth-seeking. She suggests, in her words, to “hold your identity lightly.” It is often impractical to avoid labels such as “I’m a vegan” or “I’m a Democrat”, or “I’m Republican”. Therefore it’s better to use such labels but thinking of them as just collections of beliefs rather than identities. You can say out loud, “I am x,” but you should always keep in mind what beliefs entailed by the label “x” you actually hold and what beliefs you don’t actually hold. You think of beliefs as something external to your identity that you can keep or discard when you have evidence to do so, not as flags to wave or clothes to wear, or things that make you superior to other people. Julia Galef makes a final suggestion: hold a scout identity. She suggests this because having an identity that says, “I’m the kind of person who pursues the truth” makes truth-seeking easier and more meaningful. Identities are something extremely useful to humans. Holding a particular identity makes it easier to do difficult, unrewarding things associated with that identity. If you identify as a math person, it will be easier for you to study math. Even if the practical rewards for studying math usually appear in the long term, your identity means you find the short-term actions more rewarding. Identifying as a scout enables you to keep only the best things about identities. You don’t sacrifice truth, but you can still feel motivated. Now here’s a consideration: making your identity a scout identity might still feel a bit weird. Identities are usually slowly formed, not chosen. Plus, what if, for any reason at all, you want to shed your scout identity? You might consider if choosing a scout identity still means that you are artificially putting limits on your freedom. Realistically, it shouldn’t be the case, and yet… let me know what you think. Also like, subscribe, seek the truth, tell the truth, become patrons, defeat death, conquer the stars. Discuss Benchmarking an old chess engine on new hardware 16 июля, 2021 - 10:58 Published on July 16, 2021 7:58 AM GMT I previously explored the performance of a modern chess engine on old hardware (1, 2). Paul Christiano asked for the case of an old engine running on modern hardware. This is the topic of the present post. State of the art Through an online search, I found the CCRL Blitz Rating list. It is run on an i7-4770k at 9.2 MNodes/s. The time controls are 2min+1s per move, i.e. 160s per 40 moves, or 4s per move. On the 4770k, that's 36.8 MNodes/move. The current number one on that list is Stockfish 14 at 3745 ELO. The list includes Fritz 5.32 from 1997, but on old hardware (Pentium 90. Over the years, CCRL moved through P90-P200-K62/450-Athlon1200-Athlon4600-4770k). I screened the list, but found no case of old-engine on modern hardware. • One solution may be to reach out to the CCRL and ask for a special test run, calibrated against several modern engines. • I can also imagine that the Swedish Chess Computer Association is able to perform the test directly, using their Rating List procedure. It covers 155,019 games played by 397 computers, going back to 2 MHz machines from the year 1984. They may be able to transplant an old version onto a new machine for cross-calibration. The right person to contact would be Lars Sandin. But the devil is in the details... Making own experiments In principle, the experiment should be trivial. The standard tool to compare chess engines is the command-line interface cutechess-cli. Set up old and new and get it running. Problem 1: Find an old engine with a working interface Paul and I independently favored the program Fritz from the most famous chess year 1997, because it was at the time well respected, had seen serious development effort over many years, and won competitions. The problem is that it only supports today's standard interface, UCI, since version 7 from the year 2001. So with that we can go back 20 years, but no more. Earlier versions used a proprietary interface (to connect to its GUI and to ChessBase), for which I found no converter. I was then adviced by Stefan Pohl to try Rebel 6.0 instead. Rebel was amongst the strongest engines between 1980 and 2005. For example, it won the WCCC 1992 in Madrid. It was converted to UCI by its author Ed Schröder. I believe that old Rebel makes for a similarly good comparison as old Fritz. A series of other old engines that support UCI are available for download. The following experiments should be possible with any of these. Problem 2: Configuration We can download Rebel 6 and set up cutechess like so: cutechess-cli -engine cmd="C:\SF14.exe" proto=uci option.Hash=128 tc=40/60+0.6 ponder=off -engine cmd="rebeluci.exe" dir="C:\Rebel 6.0" timemargin=1000 proto=uci tc=40/60+0.6 ponder=off -rounds 20 One can define hash (RAM), pondering, and other settings through UCI in cutechess. However, Rebel does not accept these settings through the UCI interface. Instead, they must be defined in its config file wb2uci.eng: Ponder = false Set InitString = BookOff/n Program = rebel6.exe w7 rebel6.eng The "wX" parameter sets hash/RAM: w0=2 MB, w1=4 MB,.. w9=512 MB So, we can test RAM settings between 2 MB and 512 MB for the old engine. Stockfish is typically run at its default of 128 MB. That amount would have been possible on old machines (486 era), although it would have been not common. For Rebel 6, the interface runs through an adaptor, which takes time. If we would ignore the fact, it would simply loose due to time control violations. So, we need to give it 1000ms of slack with "timemargin=1000". Now, let's go! Problem 3: Measuring the quality of a player that losses every single game I ran this experiment over night. The result after 1,000 matches? Stockfish won all of them. So it appears that Rebel is worse, but by how much? You can't tell if it looses almost every game. Rebel 6.0 has ELO 2415 on a P90, Stockfish 13 is 3544 (SF14 is not in that list yet). That's a difference of 1129 ELO, with expectations to draw only one game in 2199; and loose the rest. Of course, Rebel 6 will have more than 2415 ELO when running on a modern machine - that's what we want to measure. Measuring the gap costs a lot of compute, because many games need to be played. OK, can't we just make SF run slower until it is more equal? Sure, we can do that, but that's a different experiment: The one of my previous post. So, let's keep the time controls to something sensible, at least at Blitz level for SF. With time controls of 30s, a game takes about a minute. We expect to play for at most 2199 games (worth 36 hours) until the first draw occurs. If we want to collect 10 draws to fight small number statistics, that's worth 15 days of compute. On a 16-core machine, 1 day. Unfortunately, I'm currently on vacation with no access to a good computer. If somebody out there has the resources to execute the experiment, let me know - happy to assist in setting it up! Experiment: SF3 versus SF13 There is a similar experiment with less old software that can be done on smaller computers: Going back 8 years between new and old SF versions. The Stockfish team self-tests new versions against old. This self-testing of Stockfish inflates the ELO score: • Between their SF3-SF13 is a difference of 631 ELO. • In the CCRL Rating list, which compares many engines, the difference is only 379 ELO (the list doesn't have SF14 yet). • Thus, self-testing inflates scores. Let us compare these versions (SF3 vs SF13) more rigorously. The timegap between 2013 and 2021 is ~8 years. Let us choose a good, but not ridiculous computer for both epochs (something like <1000 USD for the CPU). That would buy us an Intel i7 Intel Core i7 4770K in 2013, and an AMD 5950X in 2021. Their SF multicore speed is 10 vs. 78 MN/s; a compute factor of 8x. We can now test: • How much more compute does SF3 require to match SF13? Answer: 32x (uncertainty: 30-35x) • How much of the ELO gap does SF3 close at 8x compute? Answer: 189 i.e. 50% of 379 ELO, or 30% of 631 ELO. Interpretation: If we accept SF as amongst the very best chess programs in the last decade, we can make a more general assessment of chess compute vs. algorithm. Compute explains 30-50% of the computer chess ELO progress; algorithm improvements explain 50-70%. Experiment method: cutechess-cli -engine cmd="C:\sf3.exe" proto=uci tc=40/10 -engine cmd="C:\sf14.exe" proto=uci tc=40/20 -rounds 100 This runs a set of 100 rounds, changing colors each round, between SF3 and SF13 with their default parameters. The time controls in this example are 40 moves in 20 vs. 10 seconds etc. The time values must be explored in a wide range to determine consistency: The result may be valid for "fast" time controls, but not for longer games. • From my experience, it is required to play at least 100 games for useful uncertainties. • Due to time constraints, I have only explored the blitz regime so far (30s per game and less). Yet, the results are consistent at these short time controls. I strongly assume that it also holds for longer time controls. Then, algorithms explain ~50% of the ELO gain for Stockfish over the last 8 years. Others are invited to execute the experiment at longer time settings. Minor influence factors So far, we have used no pondering (default in cutechess), no endgame tables, no opening books, default RAM (128 MB for Stockfish). • Endgame databases: A classical space-compute trade-off. Decades ago, these were small; constrained by disk space limitations. Today, we have 7-stone endgame databases through the cloud (they weigh in at 140 TB). They seem to be worth about 50 ELO. • The influence of opening books is small. I also suspect that its influence diminished as engines got better. • Pondering: If the engine continues to calculate with opponents time, it can use ~50% more time. Typical "hit rates" are about 60%. Thus, the advantage should be similar to 30% longer time control. ELO with time is not linear, thus no fixed ELO gain can be given. • RAM sizes (hash table sizes) have a very small influence (source 1, 2). For Stockfish, it appears to be only a few ELO. Discuss Lafayette: empty traffic signals 16 июля, 2021 - 07:30 Published on July 16, 2021 4:30 AM GMT Seeking to cross a road on the walk into downtown Lafayette, instead of the normal pedestrian crossing situation, we met a button with a sign, ‘Push button to turn on warning lights’. I wondered, if I pressed it, would it then be my turn to cross? Or would there just be some warning lights? What was the difference? Do traffic buttons normally do something other than change the lights? I clearly believe they do. They make it my turn. But they don’t send a wire into the ‘whose turn is it’ variable deep in the ought-sphere, so what exactly do they do? I suspected that this button didn’t change whose turn it was, and it felt empty and devoid of some special potence of being a traffic button. I liked to imagine that it was just a normal traffic button, but taking a more nihilistic view of its role. In which case, its nihilistic view seemed to have practical consequences! It wasn’t being as good a good traffic button while saying that it didn’t change whose turn it is. It genuinely fails to coordinate the traffic so well, because here am I unable to garner the ‘right’ to cross with confidence, and there are the drivers unsure what I’m doing. But shouldn’t a traffic button be able to do its job regardless of its philosophical commitments, or without pretending to have philosophical commitments it doesn’t have? One might say that the thing going on is that it being ‘my turn’ is a fact about everyone’s expectations. For instance, if the drivers will expect me to cross, then it is ‘my turn’. (I’m tempted to say ‘if the drivers think it is my turn, then it is my turn’, but what are the drivers then thinking?) This doesn’t seem quite right, in that the drivers themselves are asking whether this light means that it is the pedestrian’s turn, and all of us seem to be asking something about the underlying truth, not about each other. Also, if I run erratically into the road, the drivers and I may both come to expect that I am going to cross, but it still isn’t my turn. I fantasized that I had woken up in a new world which was just like the old world, but where everything was like the traffic light. I would phone the doctor’s office later to ask if it was ok to cancel my appointment this late, they would just say, ‘I’ll change what it says in this schedule’. ‘But is it ok?’ ‘I will not record your cancellation.’ ‘Should I pay you?’ ‘I am not charging you’ ‘But is that like a favor, or is it the policy? Have I wronged your medical practice? Do I owe you really? Tell me if I was bad!’ ‘I erased your name from this box on my piece of paper.’ My tentative take is that turns are real, and we created them, and traffic buttons have genuine power over them, and if a traffic button doesn’t recognize their existence it is potentially at a real disadvantage, perhaps in a similar way to how a chair maker who doesn’t recognize chairs as a thing is at a disadvantage. (To be clear, I expect philosophers have much better thought out views on this, and welcome people telling me what they are–this is me thinking aloud, not philosophical advice.) Discuss How to Sleep Better 16 июля, 2021 - 03:03 Published on July 16, 2021 12:00 AM GMT This is a cross-post from my personal blog. What is this post about? Over the summer of 2020, I slept pretty badly for a long period of time and it had pretty noticeable effects on my productivity, perceived energy levels, and well-being in general. This was mostly due to the fact that my room was under the southern roof of the house and it had around 28 degrees Celcius during "cool" summer nights which made it effectively impossible for me to get more than 6 hours of light sleep per night. In autumn, I moved to a new flat where I had to get a new bed, mattress, pillow, etc. Since I had felt the negative consequences of bad sleep pretty recently, I figured I could conduct a bit of research and use science to improve my own sleep. In the following, you can find a summary of what I found out about the right equipment and behavior to improve sleep mixed with my own considerations. One piece of framing that I want to provide before diving into the main part is about the money and effort one should be willing to spend to improve our sleep. The longer I thought about this question the more I thought the right answer should be: "a lot". Just think of the effect that good vs. bad sleep has on your life and quantify it roughly. Let's assume the difference between good vs. bad sleep results in 30 more productive minutes per day. If you value your time at the level of the minimum wage (as a conservative estimate) which is around 10€ in Germany this means you should be willing to pay up to 5€ per day. Over a year that makes 5 * 365 = 1825€. If you additionally consider all the negative side effects of bad sleep this number gets larger very quickly. If you have, for example, a mattress that is way too soft for you, your spine might suffer which can have large consequences for the rest of your life, e.g. while sitting, driving a car, or during exercise. Treating these outcomes, which are often only revealed after years and take a long time to fix, is often very expensive and emotionally draining or painful. A second example would be our perceived well-being throughout the day. I think we all know that there is a noticeable difference between a day after a good night and a bad one. One day feels slow, annoying, and unproductive while the other feels faster, more energetic, and more productive. While this is harder to quantify financially it is still something that I would be willing to put a lot of effort in to improve consistently. The right bed There are a lot of guides out there telling you how to choose the right mattress, pillow, blanket, bed frame, etc. to optimize your sleep. In this chapter, I want to give a broad overview of the different options and how confident we should be in publicly available advice. Unfortunately, though, I have to start this section with bad news. There is only very little solid and reliable research on the topic of mattresses. This Vice article gives a pretty concise summary of the state of research and the problems it has. Basically, it says that a) research about sleep is always very expensive since you have to distribute a lot of mattresses to a lot of people, b) it is therefore often funded by big mattress companies who have their own interests and the results should be approached with caution and c) private research or privately sponsored research often yields rather imprecise results such as "on average people preferred the medium-firm mattress to the firm one". But then most mattress companies have their own rating system and there is no unified measurement so results from one company won’t necessarily translate into another. The one trustworthy study that the authors found concluded that no mattress or mattress type is clearly superior to any other in general but for every individual person there are drastically better or worse mattresses. This finding mixed with my personal experience leads me to believe that it makes sense to research mattresses to improve my quality of sleep. Keeping in mind that they are not research-backed and should rather be interpreted as experience-based heuristics at best, we can look at the recommendations made by people who sell mattresses for a living. This guide gave me a good overview of the parameters that you can choose and should be aware of when buying a new one. The article’s main points are a) the parameters that matter most are your sleeping position, your weight, and your preferred firmness. Depending on them you should get a soft or firmer mattress. b) Mattresses are made of different materials. There are memory foam, latex, innerspring, and hybrid models. Each of which has its advantages and disadvantages with respect to sleep feeling and living age of the mattress. c) You shouldn't buy mattresses online. Go to a local store and test out different models. Really lie down for five minutes or longer per mattress in contention. If you don't know exactly what the parameters of your mattress are (which most people don't) you have to try it out (which is harder to do online). d) Quality comes at a price. It might not be a perfect predictor and you will find some outliers but usually, a more expensive mattress will give you a better sleeping experience. As with most products, there are diminishing returns but given that this choice will have a non-negligible impact on the average quality of sleep over your next 8-10 years you should not aim too low. There are also different fancy ergonomic pillows and blankets. In both cases, you will find wild claims about how the right choice of either can drastically improve your sleep and decrease negative conditions such as neck pain. However, once again, the research seems to be based to a large extent on privately sponsored or privately conducted research which we should treat with caution. As stated above I have bought a new mattress following all the guidelines. I think my quality of sleep and my perceived energy levels have increased since and it was worth the price already. But as I have moved to a new flat there are also many other parameters that have changed and it is hard to isolate the impact of the mattress compared to all others. I have also bought a fancy pillow that promised to be especially ergonomic for belly sleepers. Honestly, I mostly bought it out of curiosity and because I low-key wanted to show that these are mostly empty promises without tangible effect. But I was wrong. I really like the new pillow. It fits exactly to my sleeping position and my neck is very relaxed every morning. If you have neck problems after sleeping I would recommend testing out a couple of different pillows, they are astonishingly cheap (20-50 bucks) for the effect that they can have. From my perspective, the most sensible view of choosing the right mattress, pillow, etc. is that they will yield small improvements for most people but prevent large harms for some. So if your sleep is already pretty OK you can expect a small but tangible improvement. If you have neck pain after waking up or troubled sleep in general, a change of equipment might have a very noticeable positive effect on your sleep and life in general. A note on your Sleeping position: Appearantly some people believe that one sleeping position is clearly superior to all others and therefore you should actively switch to that. While there are advantages and disadvantages to all positions, e.g. certain sleep-related conditions arise less when you sleep on your back, there is no clear best position. Additionally, it is very hard to train a different position from the one that feels natural to you. So if you don't have a condition that can clearly be linked to your position (e.g. some instances of snoring) the guides recommend adapting your mattress, pillow, etc. to your position rather than changing your position. Sleep Hygiene Even though you can get measurable improvements through a change of equipment the largest improvements probably come only with behavioral changes and putting in some effort - buying a good bike certainly improves your speed but you still have to actually train to get large improvements. In this part of the post, I will talk about everything that can broadly be attributed to a change of surroundings and behavior on our quality of sleep. Biology behind Sleeping To understand a bit better why certain behaviors or environments are desirable for better sleep we first need to look at the biology of sleep. If we know how it works we can more easily improve it. Obviously, I will simplify and my neuroscience professor would not be very happy but I will try to give a concise summary of the main concepts. The Circadian Rhythm describes the 24-hours wake/sleep cycle that most humans experience. It is controlled by multiple parts of the brain one of which is the pineal gland. When the eyes receive sunlight the pineal gland does not produce melatonin and you feel awake. If it doesn't receive sunlight it does produce melatonin and you feel tired. However, this is clearly not the only mechanism controlling your circadian rhythm - otherwise, we would not experience jetlag. The Suprachiasmatic Nucleus (SCN) is a small subpart of the Hypothalamus that functions as the "master clock". It is responsible for keeping up the 24-hour rhythm independent of sunlight. This rhythm can be adapted, e.g. when we switch timezones, or forcefully changed in length but it takes time and comes with a potentially high cost to the person’s mental state. There are different stages and types of sleep. Stage one describes a transition stage where we start to relax. People awakened during this stage often deny that they have been sleeping at all. Stage two describes light sleep but people can still easily be woken up. Stage three is moderate deep sleep and usually starts 20 minutes after stage one. Your muscles are relaxed and vitals (body temperature, blood pressure, and heart and respiratory rates) fall. Stage four is the last stage and it describes very deep sleep. Your muscles are relaxed, vitals are at the lowest level and it is very hard to wake you up. This stage is only reached during the two earliest 90-minute cycles of your sleep. These 90-minute cycles are always between awake and a rapid eye movement (REM) phase or between two REM phases. During every cycle, you will go from a lighter to a deeper stage and come back to a lighter one. The REM phases increase in length over the duration of a night and they are thought to be linked to memory consolidation. The current scientific hypothesis is that you relive memories during the REM phase, strengthen important and weaken unimportant ones. Sleep stages I will try to use this knowledge to motivate parts of the following heuristics but this is not always possible as our scientific understanding of sleep is rather limited. Measuring stuff in the brain is always hard but gets harder when people can't communicate because they are asleep. Additionally, many experiments that might yield insights into sleep would involve a lot of sleep deprivation and are often not granted ethical approval. Sleep-friendly Environment In which circumstances we sleep can already have a noticeable effect on its quality. Here are some environment-related parameters that one could improve. 1. Light: As described above, your pineal gland doesn't produce melatonin when your eyes receive too much light. So if your room isn't sufficiently dark you will have a harder time falling asleep. If your window is next to a street lamp and you have a bad curtain or blind that you can't change because your landlord doesn't allow it I would recommend trying a sleeping mask. It's a bit weird to get used to but can help wonders once you do. If you have a sleeping rhythm that is compatible with daylight times I would also recommend curtains or blinds that allow in some light to be gradually woken up by sunlight in the morning. In my experience, this is a less disruptive and more pleasant way of waking up. 2. Noise: Noise can disrupt your sleep and thereby decrease the time you spend in stage three or four. However, this is primarily the case due to abrupt noises, e.g. the squeaking breaks of a car but not so much for regular noises such as the constant flow of cars on a highway as your brain quickly adapts and filters them out. If you sometimes wake up due to noise you should try to sleep with earplugs, they work wonders in hostels but can also improve your regular sleep at home. 3. Temperature: If you sleep in conditions that are too hot or too cold your sleep will be affected negatively. Most people have probably experienced the upper ends of this scale when sleeping in a very hot room in the summer or the lower end when camping with a light sleeping bag under cold conditions. On average people prefer a room temperature of 18-22 degrees Celcius. 4. Oxygen: Not much to say here. Your body needs oxygen. This doesn't stop while we sleep. If possible open a window. Honestly, I think most of us have been told these things since we were children but are either too lazy to care about them or lack the inertia to e.g. try sleeping masks or earplugs. Thus, this section is less about learning new things but rather a friendly nudge and reminder for you. If you have lacked the inertia to improve these really simple things, order the earplugs now and set yourself an alarm to open the windows before you go to bed. Behavioural Change Besides optimizing our environment we can and should also optimize our behavior w.r.t. sleep. There are tons of good guides out there, e.g. by the sleep foundation, the NHS, headspace, James Clear's blog, and many more which I will try to condense and summarize in the following. Sleep the right amount: There exist recommendations on the optimal amount of sleep but their help is limited. They recommend that young adults should sleep between 7 and 9 hours and you should adapt your length of sleep based on many conditions including your work, your health, your stress level, the amount of exercise on a given day, your caffeine intake, etc. I guess the most important information is that you should try to prevent sleeping less than 7 hours over prolonged periods of time if you are a young adult. Otherwise, this information is a bit useless as they don't give clear guidelines on how much I should shorten or extend my average amount of sleep for a given condition. In the 'Fancy Gadgets' section, I will explore whether this question can be meaningfully answered by technology. Make a sleep schedule: All the guides basically agree on two things: a) Do not accumulate sleep debt. If your regular amount is 8 hours per night you can't replace that with sleeping 6 hours for five nights and then 13 hours for two nights. Sleeping less to learn more in preparation for an exam or similar circumstance also leads to worse performance (see e.g. here). b) Don't change your rhythm drastically. If you have found a working rhythm, e.g. from 11 pm to 7 am you should not change that. This includes weekends and holidays. The more you change the less rested you will be and the more sleep you need overall. As alluded to above your SCN adapts very slowly and too drastic changes will decrease sleep quality. Get a pre-bed routine: This essentially boils down to getting yourself physically ready for bed and wind-down mentally. Possible strategies include: stop working, getting away from screens, relaxing exercises such as yoga, meditation or guided wind-downs. My girlfriend uses a sleep routine from headspace and it seems to work well for her. This seems to be different from person to person though. I usually get so bored from the wind-downs that I get annoyed by it and then can't sleep because I'm annoyed. Another option is to distract your mind and thoughts that keep you awake by listening to podcasts. Keep your bed for sleep and sex only: Your bed should be a comfort zone and not associated with anything that could cause stress or negative thoughts. If you work or watch Netflix on your bed, you might associate it with negative things or emotional experiences and thereby increase the time you need to get in the zone. While this is sound advice, whether you can actually follow it or not seems to mostly depend on your financial situation. While I was a student my rooms were so small that it was impossible to separate the categories entirely. Apply pro-sleep behavior during the day: There are plenty of things you can do during the day to improve your sleep at night. You can exercise - it will reduce the time it takes to fall asleep. I think regular intense exercise (e.g. 1h of cycling) has had the biggest positive effect on my overall quality of sleep. See the light of day and reset the melatonin in your brain. Don't eat too late. The exact reason for why it hampers with sleep quality seems to be unclear but many people report that it does. Don't drink alcohol. While it may help you to fall asleep it will increase sleep disruptions later when liver enzymes metabolize the alcohol during the night (see e.g. here). Don't smoke. It is generally terrible for your body, nicotine makes you wake up similar to caffeine and smokers have a higher risk of breathing-related sleep problems during sleep (see e.g. here). Don't drink too much caffeine, especially close to bedtime. Try to sleep only when you're tired: If you can't sleep there is no point in focusing on it too much. It will only make you feel bad for missing out on important minutes or hours of sleep while time seems to stand still. The general recommendation is that if you can't fall asleep after 20 minutes of trying you should do something else. Read a book, listen to a podcast, meditate, etc. However, you should avoid bright screens and you should be able to stop the activity whenever you feel tired again and go back to bed. Screens: While the general recommendation is to avoid screens entirely before sleeping, I think this is not always practical. Sometimes I want to read something online or as a PDF or maybe just watch a dumb YouTube video to wind down. I would recommend using programs like redshift or a native night mode on your computer. You can adjust the color brightness and the amount of red in it to simulate natural light. Since I have started using these programs I don't have screen-related sleeping problems and I have gotten so used to them that I get headaches and my eyes hurt on computers without these programs. Most modern smartphones also have some sort of night mode that you should activate. It turns the screen black and white and disables most notifications. As soon as it turns on, I just don't want to look at my phone anymore - it's astonishingly effective for me. Power naps: If you are very tired throughout the day you can do power naps. The two things that are always emphasized are a) Don't power nap too late. Optimally, you should do it after lunch but before 4 pm. b) They should not last longer than 20 minutes. Otherwise, you get into the deeper sleep phases and will feel groggy for the rest of the day. For most people, it requires a bit of training to fall asleep quickly and get back up after 20 minutes but it is manageable with a bit of training. While I was skeptical a few years ago, I now use them regularly. Whenever I feel too tired to be productive I just sleep for 10-20 minutes and return fresh. It's much preferable to an hour of low productivity after lunch in my personal experience. Fancy Gadgets There are many gadgets and apps out there that promise to improve your sleep. While I was very skeptical in the beginning I was still curious and wanted to try one out to see how well it works. I could very quickly exclude all apps that infer your quality of sleep through the motion sensors of the phone which you are supposed to lay on your mattress. While this probably provides some data, this channel seems way too noisy for me to be actually useful. Additionally, these apps don't incorporate information about your day (e.g. exercise levels) and therefore can't give you any actionable recommendations. After reading a bit further I could boil the competition down to two options: the Oura Ring and Whoop. The Oura Ring is, as the name suggests a ring that you keep on all the time and it measures blood flow and other values in your finger to infer heart rate, breathing patterns, etc. Whoop uses a bracelet and basically infers similar information from your wrist instead of your finger. While the Oura Ring is solely focused on sleep, Whoop can also be used for exercising or computing general strain throughout the day. If you want to use the gadget solely for sleep I think none of them are clearly superior. Since I recently picked up cycling again, I also wanted to make use of the exercising functions and therefore decided in favor of Whoop. This video made me curious and convinced me to test Whoop. I have been using the Whoop strap since October 2020 and so far I found two things. Firstly, the measurements and their interpretations seem to broadly align with my intuition. It detects pretty accurately when I sleep and when I wake up and most of the time when I feel well-rested the app says my recovery is high and when I feel tired the app tells me my recovery is low. However, Whoop uses heart rate variation as a proxy for recovery which is already imperfect for physical activity but definitely doesn't capture mental activity as much as it should and might thereby skew the results a bit. Secondly, whenever my intuition and whoops measurement disagree, the app is far more often correct than I am. When I wake up and feel good but the app says I'm not well recovered I usually get tired and unproductive earlier than usual. I also think that it is not a self-fulfilling prophecy, i.e. I only get tired because I expect it due to whoop, because I often forget what Whoop told me in the morning and only remember once my unproductive phase hits me. I don't think the app is a must-have. The reason why I will probably keep using it and therefore am willing to spend 0.5-1€ per day for it is two-fold: a) Just tracking the amount of sleep I get is already helpful. Even though you might have a rough estimate of the length of last night’s sleep without a sleep tracker, you probably don't have a good intuition for the last week or last month. Just having the actual numbers is already informative to evaluate your current need or see patterns. And having the app puts more attention on sleep already making it more likely that I prioritize it higher. b) The app is a good indicator of when I need to take a break. When I'm really tired and have an unproductive day, I usually feel bad for not getting things done. If Whoop says that my recovery is terrible after working long, I have to take time off. Since the app "made the decision for me" I don't feel guilty and can focus on other tasks that need less attention. After writing the original post, I found eightsleep. It's a start-up that optimizes everything around sleep and their reviews indicate that they do so quite successfully. Unfortunately, their products are only available in the US and Canada so far and too expensive for me, but when I have more disposable income, I would be curious to try them out. There is also a new (German) app called Mementor which seems to have pretty promising results in randomized controlled trials (see here). Unfortunately, though, you have to get a prescription from an actual doctor to be allowed to use the app and it is only designed for more serious problems related to sleep so I can't try it out. If you have personal experience with the app and want to share it please reach out to me. Conclusions From my research and my own experiences over the last month I find three main conclusions: Sleep is very very individual but you can improve your own sleep after finding out what your preferences are. There are no one-fits-all solutions but by thinking about what type of sleeper you are realizing your own bad habits, etc. you can improve your sleep significantly. Especially if you have problems falling asleep, disrupted sleep, or pain in a body part like the spine or neck after sleeping there is a good chance you can solve them. It makes sense to invest time and money to improve your sleep. Sleep just has a large influence on many things that are important to us. If you are well-rested you are more productive, you feel better, and you are less likely to be annoyed quickly and emotionally hurt the people you like. All of these are things we value highly and thus should be willing to invest time and money to improve. Most of this seems pretty obvious you just gotta make it a priority and do it. Most of the insights presented in the literature and summarized in this post are not revolutionary. Sleeping the right amount improves how rested you are the next day - no shit. But the fact that they seem so obvious is dangerous because you nearly feel like "you do them already because they make intuitive sense". However, at least I, find myself too often in a situation where I go to bed too late even though I know that I have to get up early the next morning or do other things that go against the classic advice. I think optimizing your sleep usually is more a problem of implementation rather than lack of knowledge. If you have trouble following the advice I can recommend to nudge yourself and thereby make your choice easier. Use technology to help you. Set yourself a time after which certain behavior is not allowed anymore or make a daily short walk part of your routine. One last note If you want to get informed about new posts you can follow me on Twitter or subscribe to my mailing list. If you have any feedback regarding anything (i.e. layout or opinions) please tell me in a constructive manner via your preferred means of communication. Discuss Generalizing Koopman-Pitman-Darmois 16 июля, 2021 - 01:33 Published on July 15, 2021 10:33 PM GMT Suppose we want to estimate the mean and variance of a normal distribution from some samples X1…Xn.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} from the distribution. It turns out that we don’t need to keep around all of the data: only the sample mean ^μ=1n∑iXi and sample variance ^Σ=1n∑i(Xi−^μ)2 are needed for the estimation problem. From those, we can calculate the entire posterior distribution of μ,Σ. The pair (^μ,^Σ) is a “sufficient statistic” for X1…Xn: a statistic which summarizes all the estimation-relevant information from the data. In principle, every estimation problem has a sufficient statistic: just let the summary be all the data. But this isn’t very useful. We really want to compress the data, pull out the actually-useful parts and throw away the rest. In the normal distribution example, our n samples of data are each 1-dimensional for a total of n dimensions of data, but the summary is only 2-dimensional, no matter how large n grows. When will an estimation problem have a fixed-dimensional sufficient statistic as the number of data points increases, like the normal-distribution estimation problem? The Koopman-Pitman-Darmois (KPD) Theorem says that, when estimating distribution parameters from IID samples, there is a fixed-dimensional sufficient statistic if-and-only-if the distribution can be expressed in exponential family form: P[X|Θ]=1Z(Θ)ef(Θ)T∑ig(Xi)∏ih(Xi) … for some vector-valued functions f,g and positive-real-valued function h. The sufficient statistic is then ∑ni=1g(Xi). For instance, for a normal distribution, the parameters Θ are (μ,Σ), and we can use g(x)=(x,x2) and f(Θ)=f(μ,Σ)=(−Σ−1μ,−12Σ−1). Intuitively, this says the “coupling” between X and Θ is only via the fixed-dimensional “summary” vectors f(Θ) and ∑ig(Xi). That’s the original KPD Theorem. We’ll see in this post that the theorem generalizes a lot. It’s not just estimation of distribution parameters from IID samples - KPD can also generalize to estimation of parameters for Markov chains, or causal models with symmetry - aka stochastic programs. The theorem also extends to causal models without any symmetry, albeit slightly weaker: a small number of variables can violate the exponential form, but (roughly speaking) the number of variables which can violate the exponential form is proportional to the summary dimension. So if we have a low-dimensional summary of lots of variables, the vast majority of variables must still satisfy the exponential form. The condition which does most of the work is sparsity: we’ll assume the conditional distribution factors according to a causal model with a sparse graph. Or, in more physical terms: direct interactions are local; long-range interactions are always mediated by short-range interactions. Conditionally independent variables are just the most extreme version of this - their conditional distribution factors according to a graph with no edges at all (i.e. there are no interactions at all). In particular, I now expect most abstractions to mostly satisfy the exponential family form, though that’s a topic for future posts. The Theorems We’ll cover three generalized KPD theorems, each with different assumptions on the structure of P[X|Θ]: • Independent But Non-Identical • Causal Model/Bayes Net • IID The first is intended as a relatively-simple case to illustrate the core ideas, before we introduce more complexity. The second is the most general theorem. The third is intended to illustrate how we can leverage symmetry to rule out the annoying “exception” variables. This is definitely not a comprehensive set of KPD generalizations; these theorems and proofs are mainly intended to illustrate the general tools and techniques by which KPD generalizations can be derived. Independent But Non-Identical KPD The simplest starting point for these theorems is parameter estimation from independent-but-not-identically-distributed variables. (If you want a concrete image in mind, picture scientists trying to estimate the values of some physical constants using a bunch of different experiments.) Theorem Let (X1,…,Xn) be continuous random variables, conditionally independent given Θ (i.e. P[X|Θ]=∏iP[Xi|Θ]). There exists a D-dimensional sufficient statistic G(X) summarizing all information in X relevant to Θ only if the distribution has the form P[X|Θ]=1Z(Θ)ef(Θ)⋅∑i∉Egi(Xi)∏i∉EP[Xi|Θ0]∏i∈EP[Xi|Θ] … where: • f,{gi} are some at-most D-dimensional vector-valued summary functions • E is a set of “exception” variables, not satisfying the exponential form • The number of exception variables |E| is at-most D Intuitively, this is like the original KPD, but some variables can be “exceptions” and not follow the exponential form - their coupling with Θ is not mediated by the summary vectors f(Θ) and ∑i∉Egi(xi). However, there can only be at-most D exceptions. When the number of variables is much greater than the summary dimension (i.e. >">n>> D), this means that the vast majority of variables must satisfy the exponential form. The summary vectors mediate the interactions with Θ of all but at-most D variables. Note that this is an “only if” theorem, not an “if-and-only-if” theorem. It does establish that a fixed-dimension sufficient statistic exists if-and-only-if the distribution takes the above form, since any distribution with the above form can use G(X)=(XE,∑i∉Egi(Xi)) as a sufficient statistic. However, that sufficient statistic has dimension at-most 2D, whereas the “only if” theorem above assumes the existence of a statistic of dimension D. So there’s a “gap” in the if-and-only-if characterization: any conditionally-independent distribution with a D-dimensional sufficient statistic satisfies the form, but given a distribution which satisfies the form, we may only be able to find a 2D-dimensional sufficient statistic rather than a D-dimensional sufficient statistic. Of course, for applications with >">n>> D, the difference between D and 2D won't matter much anyway. Proof The first key idea is the Minimal Map Theorem (also proved more elegantly here): if we want to summarize all the information in X which is relevant to Θ, then the posterior distribution (θ↦P[Θ=θ|X]) is a sufficient statistic. It’s also “minimal” in the sense that the posterior distribution can be calculated from any other sufficient statistic, therefore any other sufficient statistic must contain at least as much information (this can be proven via the Data Processing Inequality). So, given our hypothetical fixed-dimensional summary statistic G(X) (where X is a vector of all the data points), we can write F(G(X))=(θ↦P[Θ=θ|X]) … for some function F. The right-hand side is a data structure containing the values of P[Θ=θ|X] for EVERY θ-value; we can think of it as a vector indexed by θ. (In the case where θ has m possible values θ1,…,θm, any map (θ↦f(θ)) with real-valued f can be represented as a length-m vector v in which vi is f(θi).) Now, the key property underlying (this version of) the KPD is conditional independence of X, i.e. P[X|θ]=∏iP[Xi|θ]. We want to leverage that factorization, so we’ll transform the distribution into a representation we can factor. First, transform the distribution to a log-odds representation: logOdds(F(G(X)))=(θ↦lnP[Θ=θ|X]P[Θ=θ0|X]) … for some arbitrary reference parameters θ0. Next, we apply Bayes’ Rule: =(θ↦lnP[X|Θ=θ]P[X|Θ=θ0]+lnP[Θ=θ]P[Θ=θ0]) … then subtract off the prior lnP[Θ=θ]P[Θ=θ0] from both sides, since it’s not a function of X: logOdds(F(G(X)))−(θ↦lnP[Θ=θ]P[Θ=θ0])=(θ↦lnP[X|Θ=θ]P[X|Θ=θ0]) To clean up the notation, we’ll define F′(G):=logOdds(F(G))−(θ↦lnP[Θ=θ]P[Θ=θ0]). Finally, we can apply the factorization (remember, that was the point of all these transformations) and get F′(G(X))=∑i(θ↦lnP[Xi|Θ=θ]P[Xi|Θ=θ0]) The right-hand side of this expression is a sum of vectors indexed by θ. Each vector is a function of just one of the Xi, and the left-hand side says that we can express the sum in terms of a D-dimensional summary G(X). So, this is an Additive Summary Equation (see appendix below for details). In fact, it’s a particularly simple Additive Summary Equation: each fi depends only on xi, and each xi has no neighbors besides itself. The Additive Summary Equation Theorem then tells us that the equation is solvable (i.e. a D-dimensional summary statistic exists) only if lnP[X|Θ=θ]P[X|Θ=θ0]=∑i∈B(lnP[Xi|Θ=θ]P[Xi|Θ=θ0])+U(θ)∑i′∉Bgi′(xi′)+C(θ) … for some B consisting of at-most D X-indices, and some U(θ) of dimension at-most D. Exponentiating both sides and simplifying a bit, this condition becomes P[X|Θ=θ]=1Z(θ)eU(θ)∑i′∉Bgi′(xi′)∏i∈BP[Xi|Θ=θ]∏i′∉BP[Xi′|Θ=θ0] … which is the desired form. Key Takeaways The main takeaway from the theorem is the idea of “exception” variables, which circumvent the exponential family form, and the idea that the number of exception variables is limited by the dimensionality of the sufficient statistic. The main takeaways from the proof are: • Start with the Minimal Map Theorem to get the “most general” sufficient statistic • Transform to apply the factorization of the distribution • Use the Additive Summary Equation Theorem We’ll reuse this general structure in the next proof. Causal Model/Bayes Net KPD Now we address a much more general case: factorization according to a Bayes Net/Causal Model. We still imagine estimating some parameters Θ, but our data is no longer conditionally independent. Conditional on the parameters, interactions between data points are no longer non-existent. However, the direct interactions are sparse - i.e. most data points interact directly with only a few other data points (though they may indirectly interact with everything). This would be the case if, for example, we are trying to estimate parameters of some complicated physical system in real-time from a single instance of the system. Theorem Let (X1,…,Xn) be continuous random variables, whose distribution conditional on Θ forms a Bayes Net on a DAG (i.e. P[X|Θ]=∏iP[Xi|Xpa(i),Θ], where pa(i) denotes the nodes with edges directly into i in the DAG). There exists a D-dimensional sufficient statistic G(X) summarizing all information in X relevant to Θ only if the distribution has the form P[X|Θ]=1Z(Θ)ef(Θ)∑i∉Egi(xi,xpa(i))∏i∉EP[Xi|Xpa(i),Θ=θ0]∏i∈EP[Xi|Xpa(i),Θ=θ] … where: • f,{gi} are some at-most D-dimensional summary functions • E is a set of “exception” variables, not satisfying the exponential form • The set of exception variables includes at-most D variables, plus variables in the Markov blanket (i.e. parents, children and parents-of-children) of those D variables, plus children of variables in the Markov blanket. This mostly generalizes the previous theorem in the obvious way, with one major exception: we can now have a lot more exception variables. Roughly speaking, exceptions happen in “chunks”: neighborhoods each consisting of a variable, its “neighbors” in the Markov blanket, and its neighbors' children. So, in order for the theorem to say anything nontrivial, the DAG has to be sparse: each variable has to have only a few neighbors. If a single variable directly touches most of the DAG, then an exception-neighborhood around that variable could include most of the variables in the model. As long as the DAG is sparse and >">n>> D, the vast majority of variables must still satisfy the exponential form. If each variable has at-most k variables in its Markov blanket, then the number of exception variables is less than k2D. For instance, in a minimal computationally-complete circuit model (e.g. NAND gates), k=6 (two inputs per node, two children per node to allow copying, and one other parent for each child), so the number of exceptions would be less than 36D. If the number of variables exceeds the summary dimension by several orders of magnitude, this would be a drop in the bucket - the vast majority of variables would still need to satisfy the exponential form. Proof This follows broadly the same structure as the proof for the Independent But Non-Identical KPD, so we’ll speed through the parts which are the same and just focus on differences. As in that proof, we apply the Minimal Map Theorem, then apply a log odds transformation, and factor the distribution. The factorization is now P[X|Θ]=∏iP[Xi|Xpa(i),Θ], so the summary equation is F′(G(X))=(θ↦lnP[X|Θ=θ]P[X|Θ=θ0])=∑i(θ↦lnP[Xi|Xpa(i),Θ=θ]P[Xi|Xpa(i),Θ=θ0]) The neighbors N(j) of a variable Xj are no longer trivial; they are exactly the Markov blanket of Xj (i.e. parents, children, and children of parents of Xj in the Bayes net). The terms which depend on XN(j) are the factors corresponding to neighbors plus children-of-neighbors, i.e. N(j)∪ch(N(j)). The Additive Summary Equation Theorem then says that a D-dimensional summary exists only if lnP[X|Θ=θ]P[X|Θ=θ0]=∑i∈E(lnP[Xi|Xpa(i),Θ=θ]P[Xi|Xpa(i),Θ=θ0])+U(θ)∑i′∉Egi′(xi′,xpa(i′))+C(θ) … where E=N(B)∪ch(N(B)), and |B| is at-most D. Exponentiating and simplifying a bit yields: P[X|Θ]=1Z(Θ)eU(Θ)∑i′∉Egi′(xi′,xpa(i′))∏i∉EP[Xi|Xpa(i),Θ0]∏i∈EP[Xi|Xpa(i),Θ] … which is the desired form. Key Takeaways The main takeaway here is the idea that exceptional variables occur in neighborhoods, so sparsity is a necessary condition for the theorem to say anything nontrivial. On the other hand, causal models are extremely general, sparsity is ubiquitous in the physical sciences, and a sparse causal model is all we need to apply the theorem. We should thus expect exponential family distributions to show up in most places where low-dimensional summaries exist, in practice. In particular, we should expect exponential family forms for most abstractions, in practice. Another important point: the methods used in the proof are “smooth”, in the sense that small violations of the assumptions should produce only small violations of the conclusions. In other words, we should also expect distributions with approximate low-dimensional summaries, which approximately factor as a sparse Bayes Net, to approximately fit the exponential form. IID KPD When our variables have some symmetry, we can sometimes leverage it to eliminate the annoying “exception” variables. The basic requirement is that our distribution P[X|Θ] is invariant under permuting some of the variables. For instance, if we have a time-symmetric Markov Chain, then we can replace every Xt with Xt+1, and the distribution remains the same. In the case of IID variables, we can swap any two variables Xi and Xj, and the distribution remains the same, i.e. P[Xi=xi,Xj=xj,…|Θ]=P[Xj=xi,Xi=xj,…|Θ] If we can swap exception variables with non-exception variables, then we can sometimes eliminate the exceptions altogether. Theorem Let (X1,…,Xn) be continuous random variables, conditionally independent and identically distributed given Θ (i.e. P[X=x|Θ]=∏iP[X1=x|Θ]). There exists a D-dimensional sufficient statistic G(X) summarizing all information in X relevant to Θ, with D <n, if-and-only-if the distribution has the form P[X|Θ]=1Z(Θ)ef(Θ)⋅∑ig(Xi)∏iP[Xi|Θ0] … where f,g are some at-most D-dimensional summary functions. Other than eliminating the exception terms, there are two differences from the independent-but-not-identically-distributed theorem: the gi’s are all the same function, and we explicitly require D <n. When D ≥n in the earlier theorem, all variables can be exceptions, so the theorem says nothing. For this theorem, we need at least one non-exceptional variable to swap the other variables with, so we require D <n. Note that this theorem is strictly stronger than the original KPD as typically stated, which also applies to the IID case. The original KPD talks about existence of a finite-dimensional summary for an infinite stream of IID variables, whereas this version applies even to a finite set of variables, so long as the dimension of the sufficient statistic is smaller than the number of variables. Proof Since this a special case of independent variables, we can start from our independent but non-identical theorem: P[X|Θ]=1Z(Θ)ef(Θ)⋅∑i∉Egi(Xi)∏i∉EP[Xi|Θ0]∏i∈EP[Xi|Θ] Since we assume D <n, there must be at least one non-exception variable. So, without loss of generality, assume X1∉E. For any other variable i∈E, we can swap Xi with X1 and integrate out all the other variables to find 1Z′(Θ)ef(Θ)⋅g1(x1)P[X1=x1|Θ0]P[Xi=xi|Θ]=P[X1=x1,Xi=xi|Θ] =P[X1=xi,Xi=x1|Θ] =1Z′(Θ)ef(Θ)⋅g1(xi)P[X1=xi|Θ0]P[Xi=x1|Θ] … then integrate out x1: P[Xi=xi|Θ]=1Z′′(Θ)ef(Θ)⋅g1(xi)P[X1=xi|Θ0] (Minor note: we’re absorbing constants into Z(Θ) from each integral, which is why it keeps gaining primes.) For variables i∉E, we can also swap with X1 in order to replace gi with g1. Swapping and integrating as before yields ef(Θ)⋅g1(xi)P[X1=xi|Θ0]=ef(Θ)⋅gi(xi)P[Xi=xi|Θ0] Substituting both of those into the original expression from the independent but non-identical theorem yields P[X=x|Θ]=1Z′′′(Θ)ef(Θ)⋅∑ig1(xi)∏iP[X1=xi|Θ0] … which is the desired form. Key Takeaways While this is a minor improvement over the original KPD, the real point is to show how symmetry can remove the exception terms. I expect this to be an especially powerful tool in conjunction with the techniques in Writing Causal Models Like We Write Programs. That post illustrates how to turn recursively-defined functions into recursively-defined causal models. The recursive calls become exactly the sort of symmetries which could potentially be used to remove exception terms in the Causal Model/Bayes Net KPD. Takeaways Summary The original Koopman-Pitman-Darmois theorem says that a fixed-dimensional sufficient statistic exists for a stream of IID variables if-and-only-if the distribution is of exponential form. This generalizes a lot. Neither independence nor identical distribution is necessary; we need only a sparse causal model. However, this generalization comes with a cost: some “exception” variables may not follow the exponential form. As long as the system is sparse, and the number of variables is much larger than the dimension of the sufficient statistic, the number of exception variables will be relatively small, and the vast majority of variables in the system will satisfy the exponential form. The proofs should also generalize to approximations - systems with approximate sufficient statistics should be approximately exponential family, except for a few variables. We can also sometimes leverage symmetry to eliminate the exception variables. If the distribution is invariant under some permutation of the variables, then we can try to permute exception variables to non-exception variables, in which case they must follow the exponential form. Appendix: Additive Summary Equation Theorem An Additive Summary Equation is a functional equation of the form F(G(x))=f(x)=∑ifi(x) … with F and G unknown, G of dimension D <dim(f), D <dim(x). In order to say anything nontrivial about the equation, we need each f-term fi to only depend on a few x-components xj, and each x-component to only influence a few f-terms. Given some xj, we can pick out the (indices of) f-components which it influences via the expression {i:∂fi∂xj≢0} We’re also interested in the “neighbors” of xj, i.e. x-components xj′ for which some fi depends on both xj and xj′. We’ll write the (indices of) xj’s neighbors as N(j); note that we do consider xj a neighbor of itself. The post on the Additive Summary Equation shows that the equation is solvable only if we can choose a set of at-most D components of x, called xB, for which f can be expressed as f(x)=∑i:∂fi∂xN(B)≢0fi(x)+U∑i′gi′(x)+C … where the gi’s have output dimension at-most D, U is a constant dim(f) by D matrix, and C is a constant vector. In particular, we can choose gi′(x)=U†(fi′(x)−fi′(x0)), C=∑i′fi′(x0) … for some x0, with i′ ranging over terms not dependent on xN(B), i.e. {i′:∂fi′∂xj≡0}. Intuitively, this says that we can’t really say anything about terms dependent on xN(B). However, xN(B) consists of at-most D variables plus their neighbors, so in a large sparse system, hopefully most terms will not depend on xN(B). And for all the terms not dependent on xN(B), the information content can be aggregated by summation - i.e. the sum ∑i′gi′(x). Note that we may sometimes want to think about f(x) whose output dimension is uncountably infinite - i.e. f(x) returns a distribution over a continuous variable θ. In that case, the dot product U†(fi′(x)−fi′(x0)) defining gi′(x) becomes an integral ∫θU†(θ)(fi′(x)(θ)−fi′(x0)(θ))dθ; everything else remains the same. For simplicity, this post's notation implicitly assumes that θ has finitely many values, but the generalization is always straightforward. Discuss AlphaFold 2 paper released: "Highly accurate protein structure prediction with AlphaFold", Jumper et al 2021 15 июля, 2021 - 22:27 Published on July 15, 2021 7:27 PM GMT Discuss Fractional progress estimates for AI timelines and implied resource requirements 15 июля, 2021 - 21:43 Published on July 15, 2021 6:43 PM GMT This post was written by Mark Xu based on interviews with Carl Shulman. It was paid for by Open Philanthropy but is not representative of their views. A draft was sent to Robin Hanson for review but received no response. Summary • Robin Hanson estimates the time until human-level AI by surveying experts about the percentage progress to human-level that has happened in their particular subfield in the last 20 years, and dividing the number of years by the percentage progress. • Such surveys look back on a period of extremely rapid growth of compute from both hardware improvements and more recently skyrocketing spending. • Hanson favors using estimates from subsets of researchers with lower progress estimates to infer AI timelines requiring centuries worth of recent growth, implying truly extraordinary sustained compute growth is necessary to surpass human performance. • Extrapolated compute levels are very large to astronomically large compared to the neural computation that took place in evolution on Earth, and thus likely far overestimate AI requirements and timelines. Introduction Suppose that you start with1.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} that grows at 10% per year. At this rate, it will take ~241 years to get $10 billion ($1010). When will you think that you’re ten percent of the way there?

You might say that you’re ten percent of the way to $10 billion when you have$1 billion. However, since your money is growing exponentially, it takes 217 years to go from $1 to$1 billion and only 24 more to go from $1 billion to$10 billion, even though the latter gap is larger in absolute terms. If you tried to guess when you would have $10 billion by taking 10x the amount of time to$1 billion, you would guess 2174 years, off by a factor of nine.

Instead, you might say you’re ten percent of the way to $1010 when you have$101, equally spacing the percentile markers along the exponent and measuring progress in terms of log(wealth). Since your money is growing perfectly exponentially, multiplying the number of years it takes to go from $1 to$10 by ten will produce the correct amount of time it will take to go from $1 to$1010.

When employing linear extrapolations, choosing a suitable metric that better tracks progress, like log wealth over wealth for investment, can make an enormous difference to forecast accuracy.

Hanson’s AI timelines estimation methodology

Hanson’s preferred method for estimating AI timelines begins with asking experts what percentage of the way to human level performance the field has come in the last n years, and whether progress has been stable, slowing, or accelerating. In Hanson’s convenience sample of his AI acquaintances he reports typical answers of 5-10% of stable progress over 20 years, and gives a similar estimate himself. He then produces an estimate for human-level performance by dividing the 20 year period by the % of progress to produce estimates of 200-400 years. Age of Em:

At the rate of progress seen by AI researchers in their subfields over the last 20 years, it would take about two to four centuries for half of these AI subfields to reach human level abilities. As achieving a human level AI probably requires human level abilities in most AI subfields, a broadly capable human level AI probably needs even longer than two to four centuries.

Before we engage with the substance of this estimate, we should note that a larger more systematic recent survey using this methodology gives much shorter timeline estimates and more reports of acceleration, as summarized by AI impacts:

• 372 years (2392), based on responses collected in Robin Hanson’s informal 2012-2017 survey.

• 36 years (2056), based on all responses collected in the 2016 Expert Survey on Progress in AI.

• 142 years (2162), based on the subset of responses to the 2016 Expert Survey on Progress in AI who had been in their subfield for at least 20 years.

• 32 years (2052), based on the subset of responses to the 2016 Expert Survey on Progress in AI about progress in deep learning or machine learning as a whole rather than narrow subfields.

...67% of respondents of the 2016 expert survey on AI and 44% of respondents who answered from Hanson’s informal survey said that progress was accelerating.

Directly asking researchers for their timelines estimates also gives much shorter estimates than using the above methodology on Hanson’s informal survey. Overall we think responses to these sorts of surveys are generally not very considered, so we will focus on the object-level henceforth.

Inputs to AI research have grown enormously in the past

William Nordhaus reports:

First, there has been a phenomenal increase in computer power over the twentieth century. Depending upon the standard used, computer performance has improved since manual computing by a factor between 1.7 trillion and 76 trillion. Second, there was a major break in the trend around World War II. Third, this study develops estimates of the growth in computer power relying on performance rather than components; the price declines using performance-based measures are markedly larger than those reported in the official statistics.

Computation available for AI has grown enormously in the decades before these surveys. If, for example, 5-10% progress is attributed to 20 years in which computation for AI research increased 1000x, then Hanson’s extrapolation method would seem to be making predictions about years of similarly explosive growth in compute inputs until human level AI performance.

We have a number of metrics showing this massive input growth:

• AI Impacts estimates that, since the 1940s, $/FLOPS fell by 10x every 7.7 years, around 35% a year. • AI Impacts estimates that, from 2011 to 2017,$/FLOPS fell by 10x every 10-16 years, around 15% to 25% a year.
• Bloom et al. estimate that semiconductor R&D efforts have grown by a factor of 18 from 1971 to 2020, around 6.8% a year.
• Amodei and Hernandez estimate that, from 2012 to 2018, the amount of compute used in the largest AI training runs increased 10x every 11.2 months, around 1180% a year.
• AI Index Report estimates that global corporate investment in AI was $68 billion in 2020, up from$13 billion in 2015, an average increase of 39% a year.
• The World Bank estimates that world GDP has grown an average of 3.5% a year from 1961 to 2019.
• Besiroglu estimates that, from 2012 to 2020, the effective number of researchers in ML rose by 10x every 4-8 years, around 33% to 78% a year.
Extrapolating past input growth yields ludicrously high estimates of the resource requirements for human level AI performance

Hanson’s survey was conducted in 2012, so we must use trends from the two decades prior when extrapolating. We conservatively ignore the major historical growth in spending on AI compute as a fraction of the economy, and especially the surge in investment in large models that drove Amodei and Hernandez’s finding of annual 10x growth in compute used in the largest deep learning models over several years. Accounting for that would yield even more extreme estimates in the extrapolation.

Extrapolating world GDP’s historical 3.5% growth for 372 years yields a 10^5x increase in the amount of money spent on the largest training run. Extrapolating the historic 10x fall in \$/FLOP every 7.7 years for 372 years yields a 10^48x increase in the amount of compute that can be purchased for that much money (we recognize that this extrapolation goes past physical limits). Together, 372 years of world GDP growth and Moore’s law yields a 10^53x increase in the amount of available compute for a single training run. Assuming GPT-3 represents the current frontier at 10^23 floating point operations (FLOP), multiplying suggests that 10^76 FLOP of compute will be available for the largest training run in 2393.

10^76 FLOP is vast relative to the evolutionary process that produce animal and human intelligence on Earth, and ludicrous overkill for existing machine learning methods to train models vastly larger than human brains:[1]

• Cotra (2020) estimates the total amount of computation done in animal nervous systems over the course of our evolution was 10^41 FLOP. 10^76 FLOP is enough to run evolution almost a trillion trillion trillion times.
• One illustration of this is that for non-reversible computers, the thermodynamic Landauer limit means that 10^76 FLOP would require vastly more energy than has been captured by all living things in the history of Earth. Landauer's principle requires that non-reversible computers at 310 Kelvin need more than 3 x 10^-21 J for each bit-erasure. Carlsmith (2020) tentatively suggests ~1 bit-erasure per FLOP, suggesting 10^51 J are needed to perform 10^76 FLOP. About 10^17 J/s of sunlight strikes the earth, so 10^34 s ~ 10^20 times the age of the universe of 100% efficient maximum-coverage terrestrial solar energy is needed to power 10^76 FLOP of irreversible computing.
• Carlsmith (2020) estimates “it [is] more likely than not that 10^15 FLOP/s is enough to perform tasks as well as the human brain”. 10^76 FLOP is enough to sustain a 10^61:1 training compute:inference compute ratio.
• Roughly approximating Cotra (2020)’s estimates, models can be trained with one datapoint per parameter and each parameter requires 10 inference FLOPs, suggesting 10^76 FLOP is enough to train a model with 10^25 parameters.
• Extrapolating scaling laws in model performance yields enormous improvements, and historically tasks with previously flat or zero performance have yielded as models became capable enough to solve the problem at all.

Some of Hanson’s writing suggests he is indeed endorsing these sorts of requirements. E.g. in one post he writes:

For example, at past rates of [usual artificial intelligence (UAI)] progress it should take two to four centuries to reach human level abilities in the typical UAI subfield, and thus even longer in most subfields. Since the world economy now doubles roughly every fifteen years, that comes to twenty doublings in three centuries. If ems show up halfway from now to full human level usual AI, there’d still be ten economic doublings to go, which would then take ten months if the economy doubled monthly. Which is definitely faster UAI progress...Thus we should expect many doublings of the em era after ems and before human level UAI

This seems to be saying that the timeline estimates he is using are indeed based on input growth rather than serial time, and so AGI requires multiplying log input growth (referred to as doublings above) of the 20 year past periods many fold.

Conclusion

So on the object-level we can reject such estimates of the resource requirements for human-level performance. If extrapolating survey responses about fractional progress per Hanson yields such absurdities, we should instead believe that respondents’ subjective progress estimates will accelerate as a function of resource inputs. In particular, that they would be end-loaded, putting higher weight on final orders of magnitude in input growth. This position is supported by the enormous differences in cognitive capabilities between humans and chimpanzees despite less than an order of magnitude difference in the quantity of brain tissue. It seems likely that ‘chimpanzee AI’ would be rated as a lot less than 90% progress towards human level performance, but chimpanzees only appeared after 99%+ of the timespan of evolution, and 90%+ of the growth in brain size.

This view also reconciles better with the survey evidence reporting acceleration and direct timeline estimates much shorter than Hanson’s.

For the more recent survey estimate of 142 years of progress with the above assumptions the result is 10^43 FLOP (more with incorporation of past growth in spending), which is more arguable but still extremely high, suggesting evolutionary levels of compute without any of the obvious advantages of intentional human design providing major efficiencies, and with scaling much worse than observed for deep learning today.

The aggregate and deep learning linear extrapolation results of several decades still suffer a version of this problem, primarily because of the unsustainably rapid growth of expenditures, e.g. it is impossible to maintain annual 10x growth in compute for the largest models for 30 years. While they report much more rapid progress and acceleration than the Hanson survey respondents, we would still expect more acceleration in subjective progress estimates as we come closer to across-the-board superhuman performance.

1. We don’t think that anthropic distortions conceal large amounts of additional difficulty because of evolutionary timings and convergent evolution, combined with already existing computer hardware and software’s demonstrated capabilities. See Shulman and Bostrom (2012) for more details. ↩︎

Discuss

Covid 7/15: Rates of Change

15 июля, 2021 - 20:10
Published on July 15, 2021 5:10 PM GMT

Cases rose by over 60% in America this week, and we’re seeing large jumps in cases around the world. I am highly suspicious about the jump in the rate of increase, but Delta certainly seems to be the real deal, and this was well above my expectations.

I worry that recently I’ve lacked sufficient skin in the game. Everyone I personally care about is vaccinated or young enough that they don’t need vaccination, so the real sense of danger is largely gone. The worry is about the reaction to Covid, rather than about Covid itself. But that’s a very real danger, and I have back that sense of ‘oh no, things could go very wrong’ because there’s the danger that we really will blow up our way of life over all this, and go into a permanent dystopia of sorts. That’s what we need to ensure does not happen.

Thus, the bulk of this post is a numbers analysis trying to figure out what we know about Delta’s transmissibility and the effectiveness of vaccines in reducing that transmissibility, using data from a variety of sources. Others are encouraged to continue this analysis and try to get to the bottom of this.

So let’s run the numbers.

The Numbers Predictions

Prediction from last week: Positivity rate of 3.3% (up 0.4%) and deaths increase by 7%.

Result: Positivity rate of 4.8% (!) and deaths increase by 15%.

Prediction for next week: Positivity rate of 4.7% (down 0.1%) and deaths unchanged.

The null prediction is always an option, here two distinct null predictions with distinct reasoning. For deaths it’s clear that there was a reporting gap as predicted, so I do not think the death rate last week represents things getting worse, but they likely should start to get worse given Delta is deadlier and cases have stopped dropping within the required time window, and it doesn’t seem like last week’s number was too artificially high.

The case number is trickier, as there’s good reasons to think the data is distorted, either by July 4 or otherwise:

That giant spike represents going from an average of 2.6% to an average of 5.0% over two days. That’s not a thing that should happen to seven day averages. If it does, then the next five days things should continue to rise as old data cycles out for new, but that didn’t happen so far.

Perhaps July 4 was truly a superspreader event in the way that previous holidays were almost always underwhelming, as much as the previous holidays make that seem unlikely. Perhaps it’s a giant delayed data dump that didn’t include negative tests. It is hard to say. What I do know is that either things were worse than this before the jump and the low number wasn’t fully real, or the jump and new number are not fully real – one of these two numbers is misleading.

Thus, we have the standard question when a number seems to clearly overshoot, where it’s unclear where the ‘real’ number is and how fast it’s moving, so it’s unclear where it will end up. In this case, substantial real growth seems almost certain, and I definitely feel like the null prediction here is ‘chickening out’ but I’m not sure which direction to go, although I notice I’m more comfortable predicting a small reversion than an increase, while noting that such a decrease wouldn’t be ‘real.’ Thus I’m choosing a very small decline, with discussion continuing in the cases section and the Delta section, but this data doesn’t make sense.

Deaths DateWESTMIDWESTSOUTHNORTHEASTTOTALMay 27-June 252783811704562991June 3-June 97208179154312883Jun 10-Jun 163686119613142254Jun 17-Jun 235294438312632066Jun 24-Jun 305504597061861901Jul 1-Jul 74593296121281528Jul 8-Jul 145323986891451764 Cases DateWESTMIDWESTSOUTHNORTHEASTTOTALMay 20-May 2633,89034,69448,97324,849142,406May 27-June 231,17220,04433,29314,66099,169Jun 3-Jun 925,98718,26732,54511,54088,339Jun 10-Jun 1623,70014,47225,7528,17772,101Jun 17-Jun 2323,85412,80126,4566,46469,575Jun 24-Jun 3023,24614,52131,7736,38875,928Jul 1-Jul 727,41317,46040,0317,06591,969Jul 8-Jul 1445,33827,54468,12911,368152,379

Cases are up by 66%, and the positive test rate is up by 65%, which implies the number of tests was constant.

If cases were up by anything like 66%, why aren’t we running more tests? Aren’t people testing in large part based on whether they suspect they have Covid, and whether they have symptoms or known exposures, which both should be up a lot? Are we somehow supply constrained on this, despite no observations of difficulty in getting testing done on demand? How do these numbers make any sense?

Thus, I come to the conclusion that the numbers don’t make sense, and don’t belong in the same universe. If cases double test counts should rise a lot, and that not happening is super weird.

For now, I’m going to mostly ignore the test percentages and act as if the raw positive test counts are more accurate, because if there’s one thing I definitely don’t believe, it’s the reported number of negative tests. That doesn’t make any sense no matter how bad things are.

One possible way for this to kinda sorta work is that perhaps there are people who get tested in order to show a negative test, whose tests get reported every time, and people who get tested because they want to actually know if they have Covid, who mostly only report when they’re positive. Then, doubling the size of the second group doesn’t change reported test counts much? That’s the best I can come up with.

What should we make of the 66% rise in cases? How much of it might be timeshifted and in what ways? Is this a fully real rise, and should we expect it to continue? If so, what happened?

Discussion of all that will continue in the Delta section, where I attempt to reconcile all the various different data points.

Vaccinations

Given no attempts to halt the course of events, this is a strong result.

There’s a new warning on J&J shots (WaPo), because there were about 100 detected cases of an autoimmune disorder out of 12.8 million shots. There was one (1) death involved, again out of 12.8 million shots. It will be an interesting control group for the previous experiment where J&J got suspended. If we take a side effect that doesn’t matter, and treat it like it exists but doesn’t matter, does that have an impact? Versus treating it as a huge freaking deal and freaking out everyone and suspending the vaccine, which we are pretty sure had a fairly large impact the last time there was a side effect of magnitude epsilon.

New paper says that neither ‘give people fact box’ nor ‘explain how mRNA vaccine was developed and that it wasn’t too fast’ impacted vaccine hesitancy. Kudos for publishing the negative result. Why aren’t we seeing a lot more studies like this of various things one might do?

The explanation I found was that in Tennessee teenagers can get vaccinated without ‘parental consent’ and this was creating problems. Which does not seem like much of an explanation, as there’s no reason why such consent should be required or even relevant. And certainly does not explain why this applies not only to Covid-19, but to all vaccinations period. Or why they think it makes sense to strip teenagers out of lists for reminder postcards if they’re scheduled for their second shots, but focus on the other vaccines part of this, if you’re considering that this all might have a perfectly logical explanation.

Perhaps we can at least partially salvage this by taking advantage of the ‘natural experiment,’ and observing what happens to vaccination rates. Do the reminder postcards do anything? What about the other things that got suspended? What happens to the vaccine rates for other diseases? If you can’t go with ‘prevent people from getting sick and dying’ you can at least upgrade your world models.

The second day exceeds the first, so those 2.2 million appointments are likely only the beginning. It’s one thing to pass up a vaccine, it’s another to pass up the ability to participate in many aspects of life. Make no mistake. If implemented, this will work.

If America’s Delta problem gets sufficiently worse that they start bringing restrictions back, and they don’t start requiring proof of vaccination in such situations but instead once again halt life for the rest of us, I call upon all of us to find this completely unacceptable, the same way I find permanent child masking unacceptable.

In such a scenario, there are two sane choices. You can either let people do what they want, or severely restrict what the unvaccinated can do. Ending life as we know it, presumably indefinitely, shouldn’t be even potentially on the table, nor should we have any tolerance for such proposals.

Delta Variant

A post entitled Delta Variant: Everything You Wanted to Know does make a real attempt to be exactly what it says on the tin. The graphs are rather cherry-picked to make things look as bad as possible, as are a number of other discussions, but the data is all legitimate. The question of the day is now exactly how bad Delta is and making sure our models of it are right to figure out what is to come. There’s a bunch of superficially contradictory data that must be reconciled, as there usually is.

Taking stock of those data points is the logical first step.

Israeli Data

Israel offers the scariest data point, suggesting greatly reduced vaccine effectiveness.

I’ve seen the Israeli data presented in slightly different ways but this is basically what they’re reporting there. I’d like to note that the story doesn’t make sense, even on its own.

For previous strains, this is saying that vaccination wasn’t protective against hospitalization, and mostly wasn’t protective against death, once someone was infected (93% reduction vs. 93% reduction), or at least once someone tested positive. We knew even then this wasn’t remotely the case.

Then with Delta we get this gigantic drop to 64% protection against infection, but then protection against hospitalization stays at 93% and protection against death rises? So conditional on infection, this is saying hospitalization protection went from ~0% to 80%? Really?

Even the 79% number seems very very strange when looked at this way alone.

You could tell a story that justifies it. In that story, vaccine protection works 96% of the time (and there’s measurement errors), and if that happens you’re protected against severe outcomes no matter which variant you face because that wasn’t a close call, but being infected at all is a lower threshold. Before, if you were successfully vaccinated you basically never got infected (in this model), whereas now if you are vaccinated you sometimes do still get infected, but it’s never serious whereas before it never got that far in the first place. Then there’s the 4% of people for whom the vaccine doesn’t work properly, who are still at real risk. Or something like that.

As a sanity check, what happens if we ignore the reports and attempt to back out the answer from the raw data on infection numbers?

That’s not a picture one can easily read, so click the link if you want to examine it.

Reminder:

There are a lot of confounders, but let’s start with the pre-vaccine comparison.

Next, let’s do a pre-Delta comparison, from when the numbers were very low, but people were mostly vaccinated:

Consider this a ‘post-vaccination’ equilibrium. Many of the young aren’t vaccinated, whereas most of the old are, so the ratios change, but almost entirely for the youngest group. Things were mostly level before otherwise, and remain mostly level now, in roughly the same order.

Now let’s look at the last day they have data for here:

It’s unfortunate we don’t have the July 10 data, but we go with what we have. This still has to represent almost all Delta cases, and gives us apples-to-apples comparisons. What can we infer from these numbers?

Also, how does this reconcile with this graph?

Israel is young, but it’s not that young, and my response to the above graph is more like ‘those sample sizes are all absurdly small because Israel didn’t have Covid that month.’ Still seems difficult to reconcile in the details, but easy in the bigger picture.

My conclusion is mostly that this is muddled enough that I can’t draw a fixed conclusion. Especially weird is the 60-79 range. One possibility is that the vaccine needs a threshold of effectiveness to prevent infection, and it’s still mostly good enough to hold off Delta, but those with weakened immune systems are in a different situation and they are mostly very old? But the share of vaccinations in such groups is still super high compared to younger groups, and the order of these groups still seems really odd. It would, however, explain how 20-39 could be the lowest major group while 0-19 is the highest, perhaps – The kids in their 20s can still largely get vaccinated, and their immune systems are still strong, so it’s highly effective? Whereas with the 40-59s it’s less effective especially on the high end, and so on.

We should also look at case counts in Israel. On June 18 they had 1.92 cases per million, right before things started rising, on June 14 it was 65.09, for R0 = 1.97. From previous data, we can presume that when Delta was a very small portion of Israeli cases, the control system adjusted things to something like R0 = 1, so we’ll keep that number in mind.

UK Data

We can presume that Alpha hasn’t increased its absolute numbers, so at this point UK is almost all Delta. On May 25 there were 35.7 cases per million. On July 14 there were 512.9, On July 4 there were 361, so the two-week R0 = 1.19. From May 25 to July 4 we get R0 = 1.32, despite a bunch of Alpha early on, so the control system has been adjusting, but also we never see a period with an extreme R0 in the UK. In April, they had Alpha and things were stable (R0 ~ 1). For an extreme situation, let’s look at relatively early lift-off of Delta and assume a base of about 30 Alpha that doesn’t count for growth rate, and start when total cases are at double that, or 60, which is June 4. Over the next 10 days things went to 107, so let’s say that this represents Delta going from 30 to 77, before much adjustment has been made, over two cycles. That would give an estimate of R0 ~ 1.60, on the high end of the range for increased base transmissibility. It doesn’t leave that much additional room. One more cycle gets us to 131, which would be R0 ~ 1.31 for Delta alone, so presumably there were adjustments being made already then.

Transmissibility

There’s the question of how much more deadly Delta is than older strains, but I’m mostly going to ignore it because it doesn’t impact the path of the pandemic much. The question is containment versus lack of containment. If Delta is twice as deadly, that’s quite bad, but it’s an isolated question. As commentators have pointed out, the data supporting the increased deadliness is not that solidly grounded. The extra viral loads are suggestive, and the early data does look like it’s more deadly for a given unvaccinated person, but our data remains not great.

The much more key variable is transmissibility of Delta versus Alpha or the original strain. I’ve been using a 120% increase, or 2.2x (220% of original infectiousness) for Delta, and a 1.4x (140% of original) for Alpha, which continues to match estimates I’ve seen. Those all presumably refer to transmissibility among the unvaccinated and ignore the vaccinated population, and I’ve been treating the numbers accordingly. I see no reason to change that methodology, but that means that we need to reconcile our numbers with the case counts that we see.

An open question is how tight the bounds are on such numbers. If we have a model where things were previously going fine and then you ‘multiply by Delta’ then the bounds are reasonably tight. However, if you think that behaviors are adjusting in real time and seasonality causes ‘random looking’ ebbs and flows naturally, then there’s room for the data to look highly misleading, and my position at this point is if anything is closer to that second view.

Either way, your model has to make predictions whose math checks out, and that’s a good place to start.

Let’s say we accept that Delta’s ‘multiplier’ is 2.2, versus the old baseline, and Alpha’s is 1.4. Now let’s suppose vaccine effectiveness is reduced from 95% to 65%. What happens? Given current vaccination rates, with an adjustment for children, that’s about another 50% increase in the rate of infections.

That would mean that Alpha to Delta is a full effective +120% increase in the rate of infection, on top of the increase from original to Alpha, or a final factor of around 3.3. Or, alternatively, it would have a ‘base’ R0 of something like 6, with a vaccine that only reduced that for the vaccinated to 2, which would mean that even in fully vaccinated populations this would double every five days under pre-pandemic behaviors.

Set aside for the moment the question of what we would want to do about that. How does it line up with the data?

In the UK, where the Delta problem is currently largest and we are confident Delta is essentially the whole pandemic (using Our World In Data as my source), we had 36 cases/100k on May 25, and 494 on July 14, 52 days later, or just over 10 cycles, and R0 ~ 1.28. During April and the first half of May, we saw the number of cases stay roughly constant within a factor of 2, so R0 > 0.9. We could factor in some increase in vaccinations, but if Delta taking over sent R0 from 0.9 to 1.28, that’s only a 42% increase, similar to what we’d expect from Delta taking over from Alpha, minus some existing more infectious strains and some extra vaccinations, give or take behavioral adjustments and seasonality.

None of this matches the greatly reduced effectiveness hypothesis, unless you presume that behaviors are substantially adjusting during this period, but the shape of the curve isn’t suggestive of that either.

American Data

In the last week, America has seen a 66% increase in cases. As discussed above, going +10% then +20% then +66% is extreme, and can’t be explained by Delta alone. The share of Delta cases can only rise by about 30% in one week, even under extreme assumptions (e.g. from 35% to 65% or something like that, would be the theoretical limit) so a +46% effect in seven days off a 30% rise would mean Delta was at least twice as transmissible as Alpha. It also means the numbers in previous weeks would have shown a bigger problem, since the displacement of Alpha by Delta has to be gradual – there’s no way there was three times the growth in Delta this week that there was last week, because math.

Overall, though, the number isn’t crazy – if we presume that the control system had already adjusted for our vaccinations and for Alpha. If we take the +66% number seriously, and compare it to the pre-Delta situation, it’s safe to presume that we had previously stabilized under a full-Alpha situation, then +66% in a week represents a 43% rise in transmissibility from Delta versus Alpha minus any extra vaccinations, so 50-60% total, which doesn’t even leave room for the vaccines to lose effectiveness since that’s our estimate range for Delta already among the unvaccinated.

Under such assumptions, we can backchain, and it matches up with the sample data claiming that by percentage there wasn’t much Delta running around in early June, despite Delta now having taken over and now being 85%+:

An alternative calculation would be to look only at Alpha, or to compare Alpha to Delta. We have a percentage for it, so we can back out its growth in absolute numbers. It was something like 60% of cases on June 15 by this graph, and is now down to 5%. Whereas Delta went from roughly 4% to 85%.

From June 15 to July 15 is thirty days. Thirty days allows for six serial intervals. So if all of that is accurate, we get a ratio of 2.5:1, or +150% increased infectiousness, which is even higher than the +120% estimate for the full effects previously measured. We then look at absolute numbers, and see Alpha having an R0 = 0.75 over this period, versus R0 = 2.25 for Delta. Which would mean that both we’ve taken a lot more precautions recently than we did before to get things that low, and also that Delta should be doubling every four days, yet cases over the last week only rose by 66% despite starting off with the majority being Delta.

Thus I don’t believe the chart above is a representative sample – the math doesn’t add up.

One can have a hypothesis that strains crowd each other out in some sense but I don’t think the base rates are high enough for that effect to be big right now.

Then again, as I’ve been noting, if you take the numbers too seriously (as in, you don’t think there are big hidden factors and random distortions) then none of this adds up.

So where does that leave us? Can we put bounds on things?

Our lower bound should presumably be that Delta is 50% more infectious than Alpha, but that vaccine effectiveness is mostly unchanged.

Under Israeli conditions, it seems mostly safe to say that Delta is at most twice as infectious as Alpha, but that about twice is possible. This is the scariest data set.

Under UK conditions, it seems mostly safe to say that Delta is at most 75% more infectious than Alpha, and it would be difficult to get to a doubling.

Under American conditions, it seems mostly safe to say that Delta is going to be less than twice as infectious, given everything we know – the math starts to fold in on itself if we get above 75% or so, in the sense that things need to look much worse than they do. I’d put a soft bound around 75%.

We could also look at any number of other places. Israel and UK are especially picked because they’re well-vaccinated and having trouble.

Now let’s look at vaccination rates. Using Bloomberg’s ‘enough for X people’ metric, Israel is around 60%, the UK is also around 60%, and the USA is at 52.4%. That’s out of the full population, which includes children, so effective vaccination rates are somewhat higher especially for Israel (which has a younger population), and one dose is more than half of two doses, so the USA is effectively closer to 60%, and we can put Israel and the UK closer to 67% in terms of our effective percentage.

Let’s presume that in the base case, vaccinated people are 96% protected in terms of transmission. If we presume that the unvaccinated transmit at a 50% higher rate, but that the effective increase is 100% in Israel, that would give a vaccine effectiveness versus transmission of 67%. If we take the UK and presume 75% increased transmission, that implies vaccine effectiveness versus transmission of 80%. For the USA, a 75% increase in transmission would imply 76% vaccine effectiveness versus transmission.

It is possible, in theory, that this difference could be that vaccine effectiveness against transmission fades somewhat over time, and the difference here would be that Israel vaccinated earlier than other countries did.

If things were instead at our lower bound, by assumption, vaccines would remain at 96% effective.

It’s really hard to put bounds on things given all the factors we can’t account for, including control system adjustments in both directions, seasonality, and so on and so forth.

These are likely not tight bounds. There are likely a lot of behavioral adjustments involved in all of this. But it’s all very noisy, and I haven’t seen other serious attempts to figure this out. I’m encouraging everyone to take a stab at this from various angles and see what you find. There’s tons of data to work with.

If this reduced effectiveness is near those upper bounds, there is a very large problem. Even fully vaccinated populations wouldn’t be able to fully return to normal if you wanted to avoid Covid outbreaks. You’d either accept that vaccinated people and children often get Covid, and it would be fine, or you’d need to impose restrictions forever, or you’d need to get a new more effective vaccine distributed in the form of booster shots. And that’s if you got close to 100% coverage, which is not going to be happening.

Thus, the question would become, if Covid is not done with us, can we decide to be done with Covid and that life beckons, or are we actually going to kill our civilization and way of life over this despite having a vaccine that renders Covid mostly harmless?

In Other News

Fox News has generally not been as anti-vaccine as its customer base likely would have preferred, but some evidence that this may no longer be true. If I have any regular viewers reading this, can you update us?

Reasonable thread laying out different questions surrounding booster shots, which oddly still leaves out the question of whether shots can be modified to work better versus Delta. I’m confused why there isn’t more discussion about that. My presumption is that modifications wouldn’t help, which is interesting in and of itself and seems worthy of mention if true.

Deal between Israel and South Korea where Israel gives Pfizer shots now that it couldn’t use and were going to expire, gets future shots in exchange because selling things for money is evil. Post (from MR) points out that Covax’s quest to allocate vaccines equally is going to end up wasting a lot of vaccine, as many places don’t have the means to distribute the shots they’ll get.

Moderna begins a trial for yearly mRNA shot that would combine vaccines for flu, COVID-19, respiratory viruses RSV and HMPV. It turns out that not only does mRNA allow us to cure a wide variety of diseases, it lets us cure all of them at the same time, because the technology allows the payloads to be delivered together. I’d be a little concerned about short term side-effects similar to the ones with the current Covid vaccine, but my hope is that the problem can mostly be solved by proper dosing.

This is also an excellent way to give people Covid booster shots without everyone freaking out. If one shot, once a year, can deal with a wide variety of problems, that should work great, so long as the misinformation from anti-vaxxers doesn’t cause too many problems.

New Zealand didn’t secure enough vaccine shots, which is unfortunate, but is taking the AZ -> Pfizer booster path seriously, which is great.

You don’t often get to pick your allies, such as in the war on school. Or is it the war on children?

Map of such insanities:

My understanding is this applies even to private schools, although good luck enforcing that. If they’re going to mandate this now, when will it end? Are they planning on keeping things like this forever? Or even going back to the torturous ‘remote learning’ at the drop of a hat

What would it take to free our children? How bad would it have to get? Shall we run the experiment and find out?

And some really are pushing to make the pandemic restrictions permanent. Who would want such a dystopia? (post

People permanently not being able to leave home after 10pm ‘without good reason’ is the kind of thing hack writers put into young adult novels. Or at least, it used to be. Then again, perhaps there’s always been a 10% share of people who shake their fists at ‘kids these days’ and actually want to be the villains in such novels. How much of this is new?

A quarter of people wanting to close casinos and nightclubs permanently (but again, how much of that has anything to do with Covid?)? A third of people favoring permanent quarantines for international travel? Tracking everyone who enters a restaurant, again, permanently? This is still less than half of people, but a third is a lot of people.

This is the fight that is coming, even if conditions are good. They hate us for our freedom. You gotta fight for your right to party

One can still party a little too hard, even outdoors. Outdoors is much safer than indoors, but that doesn’t mean one can’t push the envelope too far:

So yes, there’s mass gathering and then there’s mass gathering, might want to not do that second one. Also, ‘cannot be ruled out’ is technically true but I think we could have gone with something a little stronger. A thousand identified cases is not a thousand cases.

Or, alternatively, instead of protesting for left-wing causes, one can protest against a Communist regime, in which case your protest is dangerous:

Remember, whether or not something spreads Covid depends on whether it is approved of by the proper cultural authorities and Very Serious People. And this is what they think about protesting against authoritarian Communism.

Discuss

Experimentation with AI-generated images (VQGAN+CLIP) | Solarpunk airships fleeing a dragon

15 июля, 2021 - 14:00
Published on July 15, 2021 11:00 AM GMT

A few days ago I found the Twitter account @images_ai, which posts AI-generated images and links to these instructions for generating your own. I started playing around with it; some of my choice picks:

The first image I generated, “sci-fi heroes fighting fantasy heroes”

“cute catboys having a party”

Someone had figured out that if you add words like “unreal engine” to your prompt, you get more realistic graphics.

so here’s “sci-fi heroes fighting fantasy heroes trending on artstation | unreal engine”

“young sorcerer heiress of a cyberpunk corporation, with dragons on her side | unreal engine”

“the cat is over the moon|unreal engine”

“Ghosts of Saltmarsh”

wikiart16384 model, “Fully Automated Luxury Gay Space Communism:2 | unreal engine:1 | logo:-1”

At this point I read a tweet suggesting that a 16:9 image ratio generally produces better results, so I switched to using that.

“solarpunk forest village | studio ghibli | trending on artstation”

“mountain expedition encounters stargate | full moon | matte painting”

“xcom fighting sectoids | matte painting”

I decided to do some systematic experimentation: picking a starting prompt (“solarpunk airships fleeing a dragon”), fixing the random seed, and varying parts of the prompt to see how they changed.

In retrospect the initial prompt wasn’t great, since I’m not sure if any of the pictures really incorporated the “fleeing” aspect, so I could have picked a more fully meaningful prompt. In general, this AI seems to be better at understanding individual words than it is at understanding sentences. Oh well.

All of these use seed 13039289688260605078 (no special significance; it’s what got randomly rolled for the first one so I kept using it), image size 640*360.

I don’t have anything interesting conclude besides what you see from the images themselves. As I was posting these on Facebook, Marcello Herreshoff kindly joined in on the experimentation, coming up with additional prompts and generating pictures with them; I’ve credited his images where they appear.

“solarpunk airships fleeing a dragon”

“solarpunk airships fleeing a dragon | Dr. Seuss” (Marcello)”

“solarpunk airships fleeing a dragon | Dr. Seuss | text:-1” (Marcello)

“solarpunk airships fleeing a dragon | paul klee” (Marcello)

“solarpunk airships fleeing a dragon | Salvador Dali” (Marcello)

“solarpunk airships fleeing a dragon | Ukiyo-e” (Marcello)

“solarpunk airships fleeing a dragon | van gogh” (Marcello)

“solarpunk airships fleeing a dragon in the style of h.r. giger”

“solarpunk airships fleeing a dragon in the style of h.r. giger | pastel colors”

“solarpunk airships fleeing a dragon in the style of my little pony friendship is magic”

“solarpunk airships fleeing a dragon | studio ghibli”

“solarpunk airships fleeing a dragon | trending on artstation”

“solarpunk airships fleeing a dragon | studio ghibli | trending on artstation”

“solarpunk airships fleeing a dragon | unreal engine | trending on artstation”

“solarpunk airships fleeing a dragon | watercolor”

“matte painting of solarpunk airships fleeing a dragon”

“matte painting of solarpunk airships fleeing a dragon | trending on artstation”

“matte painting of solarpunk airships fleeing a dragon in the style of h.r. giger | studio ghibli | unreal engine | trending on artstation”

“children’s storybook illustration of solarpunk airships fleeing a dragon”

“solarpunk airships fleeing an octopus | trending on artstation”

“airships fleeing a dragon | trending on artstation”

“airships fleeing a dragon | ps1 graphics”

“airships fleeing a dragon | SNES graphics”

“airships fleeing a nuclear blast | trending on artstation”

“airships | trending on artstation”

“solarpunk trains fleeing a dragon | trending on artstation”

“solarpunk trains fleeing a dragon | ursula vernon”

“cyberpunk airships fleeing a dragon | trending on artstation”

“cyberpunk trains fleeing a dragon | trending on artstation”

“cyberpunk trains fleeing a baby | trending on artstation”

Discuss

A closer look at chess scalings (into the past)

15 июля, 2021 - 11:13
Published on July 15, 2021 8:13 AM GMT

Introduction

I had explored measuring AI or hardware overhang in August 2020 using chess. Hardware overhang is when sufficient compute is available, but the algorithms are suboptimal. I examined the strongest chess engine of 2020, Stockfish 8, performing at 3,400 ELO under tournament conditions. When reducing compute to 1997 levels (equivalent to a Pentium-II 300 MHz), its ELO score was still ~3,000. That is an important year: In 1997, the IBM supercomputer "Deep Blue" defeated the world chess champion Gary Kasparov. With Stockfish, no supercomputer would have been required. I estimated that SF8 drops to Kasparov level on a 486-DX4 100 MHz, available already in 1994. To sum it up, the hardware overhang in chess is about 10 years, or 2-3 orders of magnitude in compute.

About a year later, in July 2021, Paul Christiano asked similar questions: How much compute would the old engine need to match the current engines? What is the influence of RAM (size and speed), opening books, endgame tables, pondering? Also, my old post gave some insights, but it can be improved by sharing the sources and making it reproducible. That's the aim of the current post (the other questions will be adressed in a later post).

Reproducing chess scaling from 2020History of PC Programs (ELO by year)

As a baseline of engine performance over the years, we plot the winner from the yearly rating list of the Swedish Chess Computer Association. Run on contemporary hardware,

• The list begins in 1984 when the program "Novag Super Constellation" reached 1631 ELO running on a 6502 CPU at 4 MHz.
• By 2005, Shredder 9 surpassed human levels on an AMD Athlon 1200 MHz.
• Today (2020), the leading engine is Stockfish 12 running on an AMD 1800X at 3.6 GHz.
Human grandmasters

To compare human grandmasters, we take the ELO over time for Kasparov and Carlsen. Carlsen's rating between 2008 and 2011 (age 18 to 21) grew from 2000 ELO to grandmaster strength, faster than any engine :-)

Deep Blue

The marker for "Deep Blue" in the year 1997 is a bit arbitrarily set to 2900 ELO. At the time, Kasparov had 2860 ELO, Deep Blue won, although close.

Stockfish 8 experiment

The main part is the Stockfish 8 experiment. How well does SF8 perform on slower PCs?

As a baseline, we need to establish its ELO at a defined speed.
1. To obtain the speed baseline, we find that SF8 makes 721 kNodes/s on an AMD Athlon 64 3500+ at 2.20 GHz.
2. We scale this linearly to 777 kNodes/s for the same CPU running at 2.4 GHz (+9%)
3. SF8 achies 3302 ELO on an Athlon 64 X2 4600+ (2.4 GHz) in the CCRL Rating List, running 40 moves in 15 minutes (one has to dig into the side details to understand which CPU name tag is which CPU. 64bit 1 CPU is the Athlon; this can also be verified with the historical version of that list.). This is an important baseline, because it cross-calibrated to dozens of other engines.
4. With that established, we can calculate the ELO as a function of kNodes/s. An average game has 40 moves. The 40 moves in 15 minutes leave 22.5 seconds per move (on average). That's 17.5 MNodes per move to achieve 3302 ELO.
5. We benchmark our own machine, on which the experiments are run. This can be done with the Stockfish parameter "bench". For simplicity, suppose our machine performs at 10 x 777 kNodes/s = 7.8 MNodes/s. That's the ballpark of recent (2020) 4-core CPUs.
6. Now we want to perform a game at 17.5 MNodes per move, on a machine running at 7.8 MNodes/s. Clearly, each move can only take 2.24 seconds. The whole 40-game match duration is: 90 seconds.
Execute the experiment

To build a ladder of SF towards slower machines, we let this version of SF8 play a set of games of 90s timecontrol versus half that (45s). The most well-established tool to compare chess engines is cutechess-cli. It is a command-line interface to play two engines (or two versions of the same engine) against each other. In the end, it nicely summarizes the results and includes a differential ELO estimate. A command may be:

cutechess-cli -fcp cmd=stockfish proto=uci tc=40/90 -scp cmd=stockfish proto=uci tc=40/45 -games 100

How bad does the version perform with less compute? In this experiment, after running 100 games, we get 14 ELO difference. That's much less than the usual statement of 70 ELO. Why is that? We can see the same effect in similar experiments down by others (1, 2): The ELO gain diminishes (flattens) at high compute. On the other hand, when we reduce compute to very low levels, the curve steepens dramatically. The full ELO loss result list from my experiment is for each halfing of compute:

ELOELO DeltakNodes/move3302 17476.43288148738.23268204369.13240282184.53205351092.33097108546.1303067273.1297753136.5280217568.327168634.1243927717.122382018.519033354.3

There is some jitter, despite increasing the number of games to 1,000 in the second half. Despite the jitter, we can clearly see the nonlinear ELO curve with compute:

The last thing we need to do is match the kNodes/move results to the old years. We may ask: In which year was the hardware available sufficient to play these kNodes/move in a usual tournament? This leaves some room for discussion. For 1997, should we choose a dual Pentium Pro 200, or a single Pentium 200 MMX? I believe it is reasonable to compare good CPUs of the time, without going overboard. After all, we're comparing chess on home computers. If we restrict it to <1000 USD CPUs for each year, we can find some SF8 benchmarking results across the web:

- AMD 5950X (2021): 71,485 kNodes/s
- Pentium III 500 MHz (1999): 127 kNodes/s
- Pentium 75 MHz (1995): 6.2 kNodes/s
- 386DX-33 MHz (1989): 1 kNode/s

There are many more such measurements found online, but for our purpose, this is sufficient. Caveats:

• Going back very far in time becomes difficult, because SF8 needed to be recompiled to reduced instruction sets to make it work; and RAM was limited in the experiment.
• It is more reasonable to match the speed to more recent years: About 200 kNodes/s in 2000, and 100 MNodes/s today. Everything before, and in between, has a factor of a few of error in its match of nodes to year.
• On the other hand, seeing benchmarks of real PCs is useful, because it encompasses uncertainties such as RAM speed.
• In reality, when considering hardware overhang for future AI, we must also ask: How well could SF8 be adapted to older hardware? Just running it unchanged will leave some performance (factor of a few?) on the table. That's a question for software engineers and compiler optimizers.

We can now bring the approximate match of Nodes/s with the years together with the other data, and present the result:

This looks quantitatively different to my first version, but is qualitatively similar.

• Again, a hardware overhang of ~10 years at maximum is visible: SF8 achieved Kasparov level in 1997
• This was only possible for contemporary PC engines of the year ~2006.
• In my old version, this was more like a 15 years gap. Back then, I had matched the speed to MIPS values for CPUs I found online.
• It is probably better to measure SF kNodes/s directly instead using a CPU speed proxy (MIPS, FLOPs, SPEC). Thus, I believe that the new figure is closer to reality.

In the next post, I will consider the other questions asked by Paul Christiano: How much compute would an old engine need to match current engines? What is the influence of opening books, endgame tables, pondering?

Discuss

Re: Competent Elites

15 июля, 2021 - 08:16
Published on July 15, 2021 5:16 AM GMT

One of the major surprises I received when I moved out of childhood into the real world, was the degree to which the world is stratified by genuine competence.

[T]hese people of the Power Elite were visibly much smarter than average mortals. In conversation they spoke quickly, sensibly, and by and large intelligently. When talk turned to deep and difficult topics, they understood faster, made fewer mistakes, were readier to adopt others' suggestions.

No, even worse than that, much worse than that: these CEOs and CTOs and hedge-fund traders, these folk of the mid-level power elite, seemed happier and more alive.

Competent Elites by Eliezer Yudkowsky

This is 100% true.

The first time I felt like I was talking to cognitive equals was at the Y-Combinator interview pool when I met a pair of Nigerians (this is not a joke) who were attempting to monopolize the entire African financial system.

The second time was when I met someone on Less Wrong who runs his own hedge fund.

Differences between geniuses are hard for non-geniuses to measure. You can understand everything that's going through the head of someone dumber than you. When someone is smarter than you, you cannot tell if they're one level above you or fifty because you literally cannot comprehend their reasoning.

General intelligence, measured relatively against your age cohort, tends to be stable over a person's lifetime. I had a teacher take me aside and tell me "I'd cure AIDs or something" when I was nine. It was that obvious.

Isn't this a contradiction? I just said you can't tell if someone is smarter than you. Then I said that my teacher (who, while smart, was not a genius) could tell that I was off the charts.

Note the qualifier "relative to your age cohort". My 4th grade teacher could tell how smart I was because I could be much smarter than a class of 4th graders while still being less smart than him.

I have observed this phenomenon in a wide variety of contexts, especially job interviews. People dumber than me can quickly tell within two minutes of conversation I'm smarter than them but it is hard for them to figure out how wide the gap is.

I once hired a technician to solder circuits. He knew my brother and I were smart because we had started a consumer hardware company in our basement. The shock came when he heard us playing an impromptu puzzle game based on our knowledge of US history. That was when he realized we weren't specialists at consumer hardware. We are that good at everything.

This, I suspect, is one of those truths so horrible that you can't talk about it in public. This is something that reporters must not write about, when they visit gatherings of the power elite.

Because the last news your readers want to hear, is that this person who is wealthier than you, is also smarter, happier, and not a bad person morally.

Competent Elites by Eliezer Yudkowsky

The smartest people tend to be ambitious.

Ambitious people are rare, so if everyone is mixed together randomly, as they tend to be early in people's lives, then the ambitious ones won't have many ambitious peers. When you take people like this and put them together with other ambitious people, they bloom like dying plants given water. Probably most ambitious people are starved for the sort of encouragement they'd get from ambitious peers, whatever their age.

The Anatomy of Determination by Paul Graham

How do you find fellow ambitious people? Not randomly. Fierce nerds are too rare to bump into by accident.

There are three ways to meet these kinds of people.

• Reaching out. This works better the younger you are (high school age is optimal) or if you have something else to offer.
• Join exclusive communities. Prestigious colleges aren't exclusive enough. Also, their entry requirements are too easily gamed. Think "at least as difficult as Y-Combinator". One way to get into exclusive communities is by volunteering. This is one way Tim Ferris got contacts when he was starting out his career. (It wasn't his only funnel. He blogged too.)
• Create media. Write a blog. Film YouTube videos. Host a podcast. Draw a webcomic. Compose your own songs. Art is my favorite for many reasons. It scales. It's humbling. It builds capital. It attracts serendipitous opportunities. It cultivates you credibility with the people you want to meet. Try out lots of different mediums. Drill down hard on whichever suits you best. The dumber you are the more you should compensate with artistic craftsmanship. The smarter you are the better you can get away with just a blog.

The best way is to combine all of these. Create art for an exclusive community. Reach out to others. Make it easy for others to reach out to you.

Discuss

Is social theory our doom?

15 июля, 2021 - 06:31
Published on July 15, 2021 3:31 AM GMT

Complexity science, and especially the dream to deeply understand social systems and build concrete predictive models (such as Asimov’s “psychohistory”), raise some slippery ethical questions. If we do develop this area of science and technology, allowing us to predict, and consequently also to purposefully leverage and manipulate our society, who gets to be at the controls? Democracy can no longer work, even conceptually, because mass opinions become merely a reflection of these control algorithms. Though individual and oligarchical leadership becomes similarly subject to such controls — though perhaps with larger uncertainty, being less predictable as per law of large numbers. Whoever is at the controls is also being controlled by the very same system — creating a feedback loop, which is not so different from how our society is already functioning.

Let’s unpack this a bit. First let’s admit that we already are, and always have been, strongly influenced by external factors. Our opinions are some fusion of the cultural trends, mass-media influencers, our family upbringing, all filtered through some inherent (genetic?) predisposition to care more about some issues than others. If we really try to dig deep, it becomes quite hard to find anything that can really be identified as “my true individual opinion.” This is similarly true of the president’s opinions, as well as that of any dictator. In this sense, there can never be some “external independent free will that controls or influences the society” — such a will is always a product of that same society, thus somehow reflecting its needs and values.

One could argue that historically, this process of formation of individuals’ opinions, and the subsequence exercise of the formed will, happens somehow “naturally” and is not consciously manipulated by anyone. Developing a predictive science that allows to engineer social behaviors would force us to make conscious choices as to how we use it. This basically amounts to someone having to take responsibility for such choices, which are presently left up to chance. And of course, with responsibility come all sorts of moral and ethical questions of how such choices “should be” made (cf. “with great power comes great responsibility”).

One specific point that bothers me is that all such reasoning comes from within our current paradigm of viewing the world: that of competition for scarce resources. From such perspective, this technology is indeed seen as a powerful tool to manipulate societies for personal gain, and a weapon to compete with other rulers. This view is not surprising, as manipulating each other is deeply embedded in our daily practice. Indeed, even when we smile at strangers on the street, most times we do it with the intention to manipulate them into liking us, in the hope of safeguarding us from possible aggression.

However, the development of a scientific understanding of social behaviors and consequences may start to shift this deeply ingrained paradigm itself. We may realize, with precise mathematical certainty, that the world is not a zero-sum game: rather than competing for a scarce resource, we can cooperate to create more of it. This way, using such technology of social engineering for maximizing personal gains may naturally lead to strategies that contribute to the common good as a side-effect. We could thus see the competition of any kind becoming fundamentally and objectively counter-productive, both for individuals and for nations.

Just as our modern social norms teach us to maximize personal gains, the ability to manipulate these norms may force us to ask what it is that we really want to “maximize.” Lacking an externally-imposed objective, we would have to face the hard problem of identifying our deeper personal motives to be alive. Thus, curiously, this power for control may guide us away from trying to control each other entirely—showing us that the true challenge lies in identifying meaningful control objectives. What would you want, if you could have anything? Facing up to that challenge, we may discover that it is honesty and transparency, rather than manipulation, that are the only tools effective at penetrating our own deepest desires.

Discuss

Lafayette: traffic vessels

15 июля, 2021 - 04:00
Published on July 15, 2021 1:00 AM GMT

This week I’m in Lafayette, a town merely twenty-three minutes further from my San Franciscan office than my usual San Franciscan home, thanks to light rail. There are deer in the street and woods on the walk from the train to town.

On this occasion at least, Lafayette doesn’t feel properly like a small town to me. I think it’s the main road. A lot of the town is spread along this single road, but the road itself doesn’t feel like its main deal is being Lafayette’s main road. It feels more focused on being an energetic transitway between somewhere and somewhere else, neither in Lafayette. Which probably isn’t even that true, since there is a perfectly giant highway also spanning Lafayette just North of it. Maybe the problem is that it’s too wide, so that the town feels like it’s tenuously accumulated on the sides of a road like plaque, rather than the road being an organic vessel of the town. Or, you know, I’m imagining things.

I seem to imagine things a lot regarding some kind of road Feng Shui (note: I know nothing about actual Feng Shui). My mind natively reads roads as conduits of some kind of ‘energy’, and tries to apply appropriate intuitive physics. For instance, if you have big flows in and out of a place, relative to the place itself, it won’t feel like its own place. It will feel like a section of a larger place. For instance, the typical random intersection in a big American city can’t be a place with its own local vibe, where you might feel like staying, because it can’t be very separate from the surrounding city that its energy (traffic?) is constantly being exchanged with. It’s just going to feel like a section of various energetic routes.

This intuitive physics is sort of like the physics of streams with leaves and debris in them. For a place to be placelike, and appealing to settle down in, it needs to have enough nooks or ponds or complications for the fast flowing streams in and out to eddy around in and slow down and let the debris be still. And this main street is a big stream running through a small place.

This is all contradicted by the frequency with which people stand in narrow thoroughfares at parties even in the face of literal physical streams of partygoers pressing against them. (My intuition on this case is that the pressure of the partygoer liquid is so high that it somehow makes sense to be stuck in the doorway, but I don’t explicitly see how this model even makes sense.) I don’t know of any pro evidence for this model, but my brain just keeps on having it.

Discuss

A cognitive algorithm for "free will."

15 июля, 2021 - 00:33
Published on July 14, 2021 9:33 PM GMT

So here's your homework problem:  What kind of cognitive algorithm, as felt from the inside, would generate the observed debate about "free will"? - Dissolving the Question

Put a glass of water in front of yourself. Now, make a decision: take a drink, or do not take a drink. Or at least imagine doing so.

When you've made your decision and acted on it, get very clear on which option you chose. Perhaps you should write it down, or at least make a mental note. Do not proceed until you've come to this point.

Now, you have a memory, one in which you seem to have made a decision of your own free will. Why does it feel that way? And why might somebody else argue that, no, in fact, you were fated to make that decision?

Let's consider the one or two main choices you made for this exercise. The first choice was whether or not to comply with the instructions. If you did, the second choice was whether or not to take a drink. Of course, there were other choices: how exactly to carry this out (in your imagination or in real life? with water, or coffee?). But we'll ignore them for now.

The process of making these major decisions took place slowly enough that you have some felt memory of the psychological operations you experienced in the course of making them. You read the instructions. Then, you may have split the future into two imagined alternatives. Some mysterious further operations that you'd have a very hard time putting into words took place. Your subconscious mind was playing a role too, but that's beyond your comprehension. But you clearly remember taking the physical actions to bring about one world state or the other.

Some aspects of this process of thought and action took place slowly and consciously enough to leave an accessible memory. Others were too fast, or two unconscious, to leave a trace.

The words of the instructions, the sense of splitting the world into two options (or not doing so), and the physical action (or lack thereof) all happened slowly enough to create a memory.

When you access that memory and reconstruct for yourself your past experience, you imagine yourself conceiving of two options - to comply with or to bypass the instructions, to drink or not to drink - and a sensation of selecting, then acting upon one of the options.

What you do not remember, because it happened too quickly or unconsciously, was the process by which that selection occurred. Or the operations underneath that. There is a physical sensation of volition that some part of your brain produces, which is perhaps a purely somatosensory feeling. Moving your arm (or not) when no external force is yanking or constraining your body physically feels distinct from being acted upon by an external force. That somatosensory sensation is the feeling of volition that you find associated with this decision in your memory.

The reason why it feels that you have made this decision of your own "free will" is that you cannot detect the overwhelming evidence against it. The mysterious, rapid, unconscious core brain activity that structures your conscious qualitative experience is unavailable to you. You have learned by imitation to describe certain somatosensory sensations in terms like will and volition. The way the instructions were written demands that you describe this reconstruction of memory in terms of "choice."

The counterargument of determinism arises both because we can observe ourselves sometimes experiencing a lack of the familiar association between the choice-making memory and the somatosensation of volition. At times, you remember making a conscious choice, but did not physically feel like you made a decision. At other times, you feel a sensation like volition, but do not remember making a conscious choice. These disturbing disconnects provide some of the missing evidence that an unconscious process is driving your psychological operations, your motor impulses, your somatosensations, and the formation of memory.

Furthermore, you realize that your ability to give an account of precisely which psychological operations led to a decision rapidly bottom out. For any given decision, after saying, "and why did you think that?" a finite (and probably small) number of times, you lose the ability to answer. Yet you infer that there must be a reason why. And you realize that reason is unavailable to conscious inspection, yet must have originated in your brain.

To really solve this homework question, you'd probably also need to explain the emotional valence of the debate, why people speculate on their own psychology in general, and why it is part of our culture. But I think this is a decent account of the mental mechanisms that produce an intuitive notion of free will, and also some evidence for an argument against it.

Discuss

Analyzing Punishment as Preventation

14 июля, 2021 - 18:12
Published on July 14, 2021 3:12 PM GMT

In the previous post we listed a number of reasons given for punishing people. The first was as preventation.

In some cases punishment can prevent the person being able to commit the crime in the future. For example it's much more difficult to commit certain crimes in prison. This is relevant if you believe people who commit such crimes once are more likely to do it again.

In this post I would like to analyze this from a game theoretic perspective.

Preventative Policies

There are actions that we do not want people to do (crimes), and there are policies we can take to reduce the chance of anyone being able to commit those crimes.

For example we don't want people to murder other people. In order to reduce the chance of murder, we could ban guns, and we could also ban all sharp implements.

However such policies have costs. If you ban guns people will find protecting themselves from bears harder. If you ban sharp implements people will find cooking a real challenge.

So in order to weigh up whether or not to impose some policy to reduce a crime, we have to consider (possible units in brackets):

1. The prior likelihood of the crime (average cases per person per year)
2. The damage caused by the crime if it occurs (per instance).
3. The amount the policy will reduce the likelihood of the crime (average cases per person per year).
4. The cost of the policy (per person per year).

If (cost of the policy) < (reduction in likelihood of crime) * (damage caused by crime) then the policy is worth implementing.

For a given crime different policies will come out differently from this calculation. Hence why some countries ban guns, but none that I know of ban sharp implements.

This assumes we can only apply the policy on a blanket level. But these factors are different for every individual. Some people are more likely to commit crimes, some people will suffer more from the policy, and so on.

Hence why every country which bans guns allows soldiers to have them - the cost of banning guns is much higher for soldiers. And many polities ban children from having sharp implements - the chance of them hurting someone with them is much higher.

By applying a customized policy to each individual you can get better total utility than you can by having a single policy that applies to all people. The cost of this is complexity, which makes designing and implementing individual policies harder, and enforcement more difficult.

In practice we usually apply a compromise, dividing up the population into a few broad, easily distinguishable categories, with a single policy for each category. We may divide things up based on age, profession, health etc. Dividing up the population based on a category which doesn't seem all that relevant can be a sign of prejudice - e.g race or gender.

Punishment and Preventation

As we gain more information about a person, we may update which policy to apply to them. For example, in countries where guns are banned, if you can prove that you are have a safe temperament, and have a need for a gun, you can get a gun licence, and you can lose that gun licence if evidence is found that you hang out on extremist sites.

If someone commits a crime, that provides us with information that he is likely to commit the crime in the future. As such some policies which previously wouldn't have been worth it now become worth it. For example, very few people commit murder, but those that do are reasonably likely to commit murder again in the future. Locking someone up for the majority of their life so they can't murder anyone is clearly an extremely excessive policy for someone who has a 0.1% chance of killing someone, but not so obviously excessive for someone with a 50% chance.

Therefore when someone commits a crime we may impose a policy on them with a very heavy cost, if the updated likelihood of them committing a crime is high enough to justify it.

From this perspective this isn't really a punishment. We would act exactly the same way whether or not the crime was intentional or completely accidental, assuming that doesn't affect the likelihood of it happening again. We would also act exactly the same way whether or not they actually committed the crime, or we found out through some other means that they were likely to commit the crime in the future. If we had an oracle which would tell us how likely a given person is to commit a certain crime, we would never update based on crimes they actually committed, and so we would never have different policies for criminals and non-criminals.

There are two ways in this gets slightly more involved. Firstly criminals is a broad, well defined category, without many people prepared to speak up for them if they are prejudiced against, and who already have records in the legal system. Hence they make an ideal category for applying different policies to than the rest of the population (and so there's a reason to differentiate actual criminals from would be criminals).

Secondly once you have decided to to apply specific policies to criminals, then the act of committing the crime signals that the cost of such policies is relatively low to the criminal (or they wouldn't have committed the crime), so it may be worth applying an even stricter policy. This would tend to make the policies you would apply to criminal stricter than those where individuals don't opt in to being in a special cased category (and hence there's a reason to differentiate intentional criminals from unintentional ones).

Punishment as Preventation in Practice

For this series I'm particularly interested in prison. So to what extent do current justice systems send people to prison as a preventative from them committing crimes?

I'm no expert at all on the legal system, but my guess is that this doesn't really drive much.

Firstly, according to my contact in British legal system, from an official policy perspective there's no provision for sending someone to prison to 'keep them off the streets'. Whilst this may play into judges thinking, it does so purely at an unofficial level - prison is about what you did, not what you might do.

Secondly, most prison sentences are relatively short. Why would we assume that someone who commits a crime today is likely to do so again tomorrow, but not in two years? If the aim of prisons was to prevent the criminal acting again, you would expect most prison sentences to be for life - or at least till they're too old to commit the crime.

Thirdly, court cases tend to revolve almost entirely around what the person under trial did, and how morally culpable he was, not around what he's likely to do in the future.

So whilst preventation might be a small part of the motivation for prison sentences, they definitely are not the main motivation.

Discuss

14 июля, 2021 - 18:11
Published on July 14, 2021 3:11 PM GMT

Summary

I’m very interested in model-based RL as a framework for how people might build AGI, in large part because I think that human intelligence is built from scratch by within-lifetime learning in a model-based RL framework. (Meanwhile I give Evolution credit for developing the learning & inference algorithms, neural network architecture, reward functions, hyperparameters, various non-”intelligence”-related activities, etc. etc.)

But a lot of people find it more natural to think in terms of “agents with desires” rather than thinking in terms of “model-based RL algorithms”.

So this is my attempt at a pedagogical discussion of how the two paradigms relate: I’ll argue that a model-based RL algorithm is kinda like a system consisting of

1. an agent with desires / preferences / goals / etc., and
2. a little demon that manipulates the desires of that agent.

This then leads into a discussion of corrigibility and wireheading. For example, I’ll argue:

If you have a model-based RL agent that does foresighted planning, it’s generally “trying to do something”, but that thing it’s trying to do is generally not “maximize the reward signal”.

This parallels the by-now-familiar discourse on inner alignment, but it's not identical because most of the existing discourse is talking about model-free RL instead.

I’ll also talk about some aspects of how this seems to work in humans, which requires going beyond that simple picture, including a system for real-time-editable desires, and the possibility / inevitability of non-self-consistent desires.

Let’s dive in!

TerminologyWhat do I mean by “model-based RL”?

In My AGI Threat Model: Misaligned Model-Based RL, I offered this (somewhat oversimplified) diagram of what I'm thinking about:

“Model-based RL” is a big category with blurry boundaries, I guess. (I don't keep up with the RL literature.) But when I talk about a model-based RL agent, I mean the following:

• The agent builds a predictive model of itself and its world based on experience of actions and observations. (“If I drop this ball it’s gonna bounce”.)
• The agent also builds a predictive model of both its immediate rewards and its longer-term reward expectations (a.k.a. value function), by combining the above model with a lifetime of experience of rewards. (“I hate stubbing my toe”.)
• …And in particular, the agent can not submit arbitrary hypothetical queries to the ground-truth reward function; it can only learn about the reward function by observing it in action. (“I thought I wouldn’t like watching this movie a second time, but actually I liked it even better!”)
• The agent takes actions that are predicted to lead to high future rewards according to its models. (“I moved the chair so that I won’t stub my toe on it”.)
• These bullet points are all separate things written into the source code. Like, maybe there’s a big complicated "world model'' data structure, and a big complicated "value function" data structure. Maybe we’re at a loss to interpret these data structures, but at least we can point to each of those data structures in RAM address space.

We can contrast this with model-free RL. There, foresighted planning might or might not happen, depending on the intricate details of its network architecture. And if it happens, the goal might or might not align with the reward function. This is by now a very familiar story on this forum—see dozens of posts about mesa-optimization, which Vladimir Mikulik pithily summarized with the slogan “Utility ≠ Reward”.

By contrast, in this kind of model-based RL, foresighted planning is part of the human-created architecture, and definitely happens, and we have at least a high-level idea of what that entails. But anyway, as I’ll argue below, Vladimir’s slogan “Utility ≠ Reward” still applies.

More terminology:  Why I'll be talking about "desires" rather than "goals", "objectives", "utility functions"

Without getting into details (I’ll get back to this in the last section), I’ll use the term “desires” to encompass everything that leads to an agent doing one thing rather than another, when both options are viable and under consideration. So “desires” includes everything from “I want to get out of debt” to “I can’t resist another cigarette” to “I like this song”. It includes both conscious and unconscious desires, endorsed and unendorsed, etc.

I want you to think of desires as being mechanistically part of the agent’s cognition. I’m not talking about the behaviorism perspective, where I am merely acting “as if” I had certain desires. These desires correspond to actual things in my brain algorithm.  Again, see the final section below for more discussion.

I’m using the term “desire” rather than “goal” or “objective” or “utility” because:

1. Terms like goal / objective / utility have a bit more of a connotation that we’re specifically talking only about desires concerning future states, not desires about process / deontology / etc., let alone the non-consciously-endorsed “desire” to keep mentally replaying the song that’s stuck in my head;
2. Terms like goal / objective / utility have a bit more of a connotation that the desires are coherent, self-consistent, well-defined, etc. For example, it seems sorta reasonable to me to say “I desire to get an early start to my day, but I also desire to keep my head on the pillow”. It would seem a bit weirder if I replaced the word “desire” in that quote with “goal” or “I assign a high utility”.
3. The term “utility function”, in particular, is traditionally associated with the behaviorism / revealed preferences perspective I mentioned above—although I suppose not everyone uses it that way.

Then a desire-driven agent is one that makes decisions on the basis of its desires. As an important special case, such agents can make decisions using foresighted planning to bring about desirable future states of the world.

A relation between a model-based RL agent and a desire-driven agent [simplest version]

So here’s a starting sketch of how you might build a reward-maximizing agent out of a desire-driven agent.

We take a desire-driven agent (call her “DEDRA”) and initialize her with no desires whatsoever. Then we add in a little demon that can get inside DEDRA’s head and edit her desires.

The demon has access to the reward signal, and the demon has access to DEDRA’s evaluation of the extent to which she is fulfilling her desires. The demon’s goal is to edit DEDRA’s desires such that the extent to which she desires something matches the corresponding total reward signal.

So if there’s a scenario for which DEDRA has “10 points of desire”, but when it happens the reward signal says “-5 reward points”, the demon notices the discrepancy and edits her desires to try to reduce the discrepancy.

Also, the demon is a simpleton. It doesn’t understand the world, or what determines the reward function, or what DEDRA’s desires actually mean semantically. So the demon has to update the desires using simple rules of thumb. For example, when it sees a discrepancy, maybe it looks at whatever DEDRA is thinking about right now, and says to itself “Hey, maybe that part of the world-model has something to do with the discrepancy?” So it tweaks the desirability of things involving that bit of the network. Or maybe the demon just does supervised-learning-via-backprop on the “desires” network, or whatever.

Another way to look at the demon’s role is: the reward function is a thing in the world, the desires are defined on DEDRA’s world model, and the demon is continually trying to match the two up.

So anyway, now we have a relation between a reward-maximizing agent and a desire-driven agent: the system as a whole (DEDRA + the demon) is a reward-maximizing agent, and one subcomponent of the system (DEDRA) is a desire-driven agent.

Then here’s the translation guide back to the world of model-based RL:

Here, “value function” is the (learned) expected sum of future rewards, and “reward model” is the (learned) prediction of rewards on any particular step. (As in MuZero, it’s helpful to have both; otherwise if you do an imagined roll-out, and you see the value suddenly jump downward, you don’t know whether this is an inconsistency in the value function, or whether there’s an expected reward for that step. Remember from above, I’m assuming we don’t have the ground-truth reward function; we have to learn it.)

You can think of the demon as providing, let’s call it, “normative grounding”. DEDRA can do a lot on her own—like she can do a hypothetical roll-out, find an area where her value function and reward model are not self-consistent, and update one to match the other. But only the demon can systematically shift DEDRA’s desires towards particular things in the world.

Why on earth would we want to make an agent this way?

“Want to”? I don’t want to! I mean, this is just awful!

For one thing, by instrumental convergence, DEDRA will by default sabotage the demon as soon as she can. Unless, of course, the demon can manipulate her in such a way that she likes and endorses the demon—a.k.a. corrigibility, see below. (Other options seem less promising to me, namely (1) prevent her from becoming aware of her situation, or (2) protect the demon from DEDRA via cybersecurity.)

For another thing, it’s hard to see how we’re going to be sure that the demon is doing its job properly, or to know how much progress it’s made. After all, the demon is using imperfect rules of thumb, and after a while the desires evolve into some horribly complicated mess, etc.

So this setup is just awful. Why on earth would we do this? In my view, the main reason to talk about this paradigm is:

People might make agents following this general paradigm because it would actually work for making a powerful goal-directed AGI, and I’m not sure we’ll come up with any other ideas.

The world is complicated. Presumably the only practical way to make an AI that understands the world is via a learning algorithm—one that finds patterns in the inputs, and then patterns in the patterns, etc. So by default the world-model will be a big mess of unlabeled items referencing each other. And then we need to define the agent’s desires in terms of this world-model. Somehow.

Well, this is one way to do that.

This method apparently works in humans—as I discussed here, I think the human brain (mainly parts of the neocortex) learns a world-model within our lifetimes from scratch, and the limbic system & brainstem & hypothalamus are involved in both calculating how desirable different plans / actions / thoughts are and updating those calculations to better match ground-truth calculations (more on which below), and several mechanisms involving dopamine ensure that we always think the most-desirable-to-us thoughts and execute the most-desirable-to-us plans.

So anyway, the point is, it seems to actually work. It creates powerful agents that understand the world and get things done. And there’s lots of researchers trying to do this kind of thing—cf. MuZero and the entire field of neuroscience, among other examples. We need to be considering the possibility that they’ll succeed.

As mentioned above, DEDRA’s desires are defined on her world-model—they need to be defined in terms of concepts that DEDRA can currently conceive of. (See related discussion at The Pointers Problem and My AGI Threat Model.) It is possible that DEDRA will (eventually) be self-aware, and be aware of the existence of the demon, and be aware of the existence of the reward function. Therefore it is possible for DEDRA to wind up with a desire to “maximize my reward function” (wireheading), or “protect and help the demon” (corrigibility), or “sabotage the demon” (non-corrigible goal preservation), and so on. We turn to these possibilities next.

I’ll distinguish two things:

• Weak wireheading drive: “I want a higher reward, other things equal.”
• Strong wireheading drive: “I want a higher reward, and this is the sum total of everything that I care about in the universe.”

In the human case, we naturally equate a wireheading drive with hedonism. I’m not sure that’s quite right—the various interoceptive feelings of pleasure, reward, etc. have a complicated relationship with the actual RL reward signal, I think—but even if it were right, it would suggest that (almost) all humans have a weak wireheading drive but not strong wireheading drive. We want to feel good, but we generally care at least a little bit about other things too.

This is what I would expect from the heuristic update-on-RPE procedure (a.k.a. demon desire-editing algorithm) that I suggested above. When we get an unanticipated reward, I suggested that maybe the demon looks at whatever we happen to be thinking about at the time, and sets those things to be more desirable. Maybe sometimes “the reward signal itself” would be on our mind, and the demon would assign some credit there, so we would come to desire the reward signal itself. But there are other things in the world-model that are getting credit too—food, friendship, the knowledge that I just stubbed my toe, and so on.

So, will a wireheading drive appear or not? And how strongly? Some relevant considerations include:

• Timing—If credit goes mainly to signals that slightly precede the reward prediction error, then the reward signal itself is not a great fit.
• “Finders Keepers”—Once you have a way to accurately predict some set of reward signals, it makes the reward prediction errors go away, so the demon stops running for those signals. So the first good reward-predicting desire gets to stick around by default. Example: we learn early in life that the "eating candy" concept predicts certain reward signals, and then we get older and learn that the "certain neural signals in my brain" concept predicts those same reward signals too. But just learning that fact doesn't automatically translate into "I really want those certain neural signals in my brain". Only the demon can make a thought appealing, and if the rewards are already being predicted competently, then the demon is inactive. (This is related to blocking in behaviorist psychology, and to “explaining away” in Bayes nets.)
• Priors—There may be some kind of bias to assign credit to predictive models that are simple functions of sensory inputs, when such a model exists, rather than functions of abstract semantic understanding, other things equal. (I'm thinking here of the relation between amygdala predictions, which I think are restricted to relatively simple functions of sensory input, versus agranular prefrontal cortex predictions, which I think can involve more abstract knowledge. I'm still kinda confused about how this works though, as discussed here.)

For AGIs, we’ll probably want to do other anti-wireheading interventions too, like (somehow) use transparency to find "the reward signal itself" in the world-model and manually lock its desirability to “neutral”, or whatever else we can think of. Also, if (as I expect) the failure mode in question is “weak wireheading drive” not strong—i.e., the desire to wirehead is trading off against other things it cares about—then hopefully conservatism (section 2 here) can help prevent catastrophe.

I note that even experts sometimes sloppily talk as if RL agents make plans towards the goal of maximizing future reward—see for example Pitfalls of Learning a Reward Function Online.

I still don’t think it’s a good idea to control a robot with a literal remote-control reward button; I just don’t think that the robot will necessarily want to grab that remote from us. It might or might not want to. It’s a complicated and interesting question.

Self-aware desires 2: corrigibility

As a human, I’m well aware that my hypothalamus & brainstem can edit and erase my deepest desires. Sometimes I don’t care. Like, I prefer chocolate ice cream over strawberry ice cream. If someday in the future I find myself preferring strawberry ice cream, then, well, OK! My tastes will evolve! Such is the human condition! It’s lovely! But other times I care very much. If someday my hypothalamus & brainstem make me gradually stop loving my children, that is not cool.

By the same token, if DEDRA is aware of the demon, she could wind up with desires related to the demon’s activities. Since goal stability is a convergent instrumental value, I expect that she will by default be motivated to sabotage the demon. However, it’s also possible that she specifically comes to like and endorse the demon and its activities. That’s (one aspect of) corrigibility, and that’s what we want for AGI safety.

If we get the reward function just right, then maybe we can get corrigibility automatically—the demon’s edits will assign high desirability to the concept of “the demon continuing to edit unperturbed”. I suspect that we’d need transparency tools to get such precise rewards. Alternatively, if we do have such transparency tools, we could just cut out the middleman, and directly locate the concept of “the demon continuing to edit unperturbed” in the world-model, and just manually set its desirability to “super-duper-desirable”.

Variation: The demon can edit “assessment functions” that serve as inputs to desirability

A cool variation is that instead of editing desires directly, the demon can edit a set of perhaps dozens of “assessment functions”, which then get fed into the real desirability-calculating function. The latter is now a simple legible function of those dozens of variables—all the complexity was pushed out of the desirability function and into the assessment functions.

I think this happens in humans and animals. As discussed in Big Picture of Phasic Dopamine, I think when you entertain a plan or idea or thought, various parts of your brain (amygdala, hippocampus, agranular prefrontal / insular / cingulate cortex, ventral striatum) calculate a whole array of anticipated physiological consequences of that plan: “If I do this, should I tense up? Should I salivate? Should I laugh? Should I cringe?” Each of those “assessments” is trained from experience by supervised learning (a.k.a. updated by the demon). Then the hypothalamus and brainstem take these assessments as inputs into the (now comparatively simple) desirability function. Importantly, they can also vary that desirability function in a situation-dependent way. For example, one of the assessment functions might be “how probable is it that this plan will result in eating lots of food?”. When you’re hungry, the hypothalamus adds extra desirability points to plans that score highly on that assessment. When you’re over-full, the hypothalamus subtracts desirability points from those same plans.

(It’s perfectly possible to directly learn that “eating when hungry is good, eating when full is bad” by pure rewards, without this more complicated two-layer system. It’s just slower, and requires more trial-and-error. See inner alignment in salt-starved rats for a concrete example of how this two-layer setup enables adaptive behavior without trial-and-error.)

I really like this setup as a generically good idea for AGI safety—in fact I was thinking about this kind of thing long before I realized that the brain works this way. (Certain ML systems work vaguely this way too, I guess.) Its two selling points are transparency and control.

The “transparency” part is, I think, self-explanatory. To be clear, I don’t have a particularly great plan for what the assessment functions should be, or how to train them, so as to help with transparency. It just seems like a good direction.

The “control” part is: When you’re driving a car, it is a critically important safety requirement that when you turn the steering wheel, the wheels respond instantaneously. By the same token, it seems like a critically important safety requirement for the human supervisor to be able to change an AGI’s goals and motivations instantaneously by pressing the appropriate button—even if the AGI is making complex abstract opaque plans about the distant future. Again, I don’t have a very specific plan here, it just seems generally promising.

Partially-grounded reward signals

There's an aspect of this scheme that I want to call out for clarity.

We calculate reward prediction error by subtracting reward prediction from reward. In textbook RL, these two terms represent a clean division:

• The reward is 100% grounded in the world, independent of the agent’s world-model and beliefs
• The reward prediction is the opposite—it’s purely a learned function of the agent's beliefs

In the two-layer setup here, the latter is still true, but the former is not true—instead, the reward signal is (in general) a messy function that mixes together grounded information from outside the world-model and learned information from inside it.

For example, if you have the idea "I'm gonna eat a salty cracker", then per the diagram above, there's an Assessment Function that says "this plan will result in eating salt". So far, this is a learned pattern in the world-model. Then there's a second step where the brainstem says, "gee, I’m kinda low on salt, so salt would be metabolically helpful right now", so it sends a positive reward for thinking that thought. This step is grounded—the brainstem-hypothalamus system gets metabolic status information straight from the body, not from the world-model. In fact, the brainstem has its own whole sensory-processing system (superior colliculus etc.), parallel and independent of the neocortex’s sensory-processing system, for getting grounded information that doesn't rely on learned representations. (Well, the two sensory systems do interact in various ways—interesting story, parts of which I still don’t entirely understand.)

I bring this up because a model-based RL AGI, like a human, will presumably need many reward signals per second in order to think thoughts and so on. But I expect that real ground-truth reward signals (e.g. from human feedback) will be relatively rare and expensive. Purely learned reward signals (e.g. as in Deep RL from human feedback) are the opposite extreme—they’re entirely dependent on the accuracy of certain learned models. It seems to me that this kind of partially-grounded reward function is maybe a nice compromise / alternative paradigm.

More on “desires”

I’ve been a bit vague about what “desires” are. I’m really thinking of a human-inspired definition along the lines of:

A “desirability function” is a function that takes a possible plan / scenario / thought and returns how “desirable” it is to make that plan / think that thought / etc.

To be clear, in my model, the brain always chooses a more desirable plan over a less desirable plan, when both are available options. I’m not trying to make some snarky point about human nature here; I mean this to be almost tautological, assuming that the brain algorithm is working properly. So if someone says to me “I didn’t want to exercise, but I did it anyway”, I would say that they must have actually found a way to think about the plan of exercising such that it was in fact a “desirable” plan on net. (See (Brainstem, Neocortex) ≠ (Base Motivations, Honorable Motivations) for much more on that.)

The demon inevitably makes somewhat-incoherent desires

The thing is, it’s a complicated world. There’s no law ensuring that the demon’s updates create a perfectly coherent, self-consistent set of desires. We could thus have one course of action, and when we “think about it in a certain way” it has high desirability, and when we “think about it in a different way” it has low desirability. This is just an inevitable part of the paradigm. DEDRA is not omniscient; there are relations and dependencies between different parts of her world-model that she hasn’t thought through. But the demon still has to tinker with her desires. It’s bound to result in something incoherent. I think there can be dynamics that make it gradually less incoherent over time (e.g. finding pareto-improving ways to reduce internal conflict, as I discussed here; or incompatible desires outright sabotaging each other, as I discussed here), but it’s no sure thing.

I guess this discussion probably has something to do with An Orthodox Case Against Utility Functions and logical induction and Jeffrey-Bolker and so on, but I’ll refrain from commenting further: I’m not an expert on that, and anyway this post is long enough already.

Discuss

Essentialness of Data

14 июля, 2021 - 06:40
Published on July 14, 2021 3:40 AM GMT

In discussing online privacy, people will sometimes say things like " if the data isn't required to operate, you don't need it." If I can turn on tracking protection, stop sending data, and nothing breaks, then clearly it wasn't needed, right? But consider these three cases:

• Learning from implicit feedback: dictation software can operate without learning what corrections people make, or a search engine can operate without learning what links people click on, but the overall quality will be lower. Each individual piece of information isn't required, but the feedback loop allows building a substantially better product.

• Incremental rollouts: when you make changes to software that operates in complex environments it can be very difficult to ensure that it operates correctly through testing alone. Incremental rollouts, with telemetry to verify that there are no regressions or that relevant bugs have been fixed, produces better software. Even Firefox collects telemetry by default.

• Ads: most websites are able to offer their writing for free, without a paywall, because they can get paid for showing ads. Collecting more data makes ads more efficient, which makes them more profitable for the sites, which translates into more competition to provide users with things to read. (more)

Instead of pushing for "don't collect data", I think it would make a lot more sense for advocates to push for "only collect data privately" and work to make that easier (carrot) or mandatory (stick). None of these uses require individual level data, they're just easiest to implement by sending all of the data back to a central server and processing it there.

(What does "private" mean? Ideally it means that no one reviewing the data can tell what your, or any other individual's, contribution was. This is formalized as differential privacy, and is typically implemented by adding noise proportional to the maximum contribution any individual could have. In some cases k-anonymity may also provide good protection, but it's trickier. And this is only the beginning; privacy researchers and engineers have been putting a lot of work into this space.)