Вы здесь

Сборщик RSS-лент

Please take a survey on the quality/impact of things I've written

Новости LessWrong.com - 29 августа, 2020 - 13:39
Published on August 29, 2020 10:39 AM GMT

If you’ve read anything I’ve written on the EA Forum or LessWrong, I’d really appreciate you taking this brief, anonymous survey. Your feedback is useful whether your opinion of my work is positive, mixed, lukewarm, meh, or negative. And remember what mama always said: If you’ve got nothing nice to say, self-selecting out of the sample for that reason will just totally bias Michael’s impact survey.

Don’t Panic![1]

I plan to use people’s responses as inputs - rather than definitive answers - in my ongoing efforts to plan my career and improve in various ways. And I’ll combine these inputs with a lot of other inputs.

Thus, you shouldn’t feel that this is uncomfortably high-stakes, nor that you should only take the survey if you’re really confident in what you’d say. You can just provide any tentative thoughts you have, and I can be responsible for working out how much weight I should give them, whether and how they should affect my decisions, etc.

(This is a good division of labour, as I know more about myself and the context of my work than you do, but you have the advantage of existing outside of my swirling vortex of alternating imposter syndrome and overconfidence.)

I’ve also added a comment below with more info on why I’m running this survey, but you’re very welcome to answer the survey without reading that comment.

  1. The Douglas Adams reference, not the Coldplay song. ↩︎



Discuss

Updates and additions to "Embedded Agency"

Новости LessWrong.com - 29 августа, 2020 - 07:22
Published on August 29, 2020 4:22 AM GMT

Abram Demski and Scott Garrabrant's "Embedded Agency" has been updated with quite a bit of new content from Abram. All the changes are live today, and can be found at any of these links:

Abram says, "I'm excited about this new version because I feel like in a lot of cases, the old version gestured at an idea but didn't go far enough to really explain. The new version feels to me like it gives the real version of the problem in cases where the previous version didn't quite make it, and explains things more thoroughly."

This diff shows all the changes to the blog version. Changes include (in addition to many added or tweaked illustrations)...

Changes to "Decision Theory":

  • "Observation counterfactuals" (discussed in the counterfactual mugging section at the end) are distinguished from "action counterfactuals" (discussed in the earlier sections). Action counterfactuals are introduced before the five-and-ten problem.
  • The introduction to the five-and-ten problem is now slower and more focused (less jumping between topics), and makes the motivation clearer.
  • Instead of highlighting "Perhaps the agent is trying to plan ahead, or reason about a game-theoretic situation in which its action has an intricate role to play." as reasons an agent might know its own action, the text now highlights points from "Embedded World-Models": a sufficiently smart agent with access to its own source code can always deduce its own conditional behaviors.
  • ε-exploration and Newcomblike problems now get full sections, rather than a few sentences each.
  • Added discussion of "Do humans make this kind of mistake?" (Text versions only.)

Changes to "Embedded World-Models":

  • "This is fine if the world 'holds still' for us; but because the map is in the world, it may implement some function." changed to "... because the map is in the world, different maps create different worlds."
  • Discussion of reflective oracles now gives more context (e.g., says what "oracle machines" are).
  • Spend more time introducing the problem of logical uncertainty: emphasize that humans handle logical uncertainty fine (text versions only); say a bit more about how logic and probability theory differ; note that the two "may seem superficially compatible, since probability theory is an extension of Boolean logic"; and describe the Gödelian and realizability obstacles to linking the two. Note explicitly that "the 'scale versus tree' problem also means that we don’t know how ordinary empirical reasoning works" (text versions only).

Changes to "Robust Delegation":

  • Introduction + Vingean Reflection:
    • Introduction expanded to explicitly describe the AI alignment, tiling agent, and stability under self-improvement problems; draw analogies to royal succession and lost purposes in human institutions; and highlight that the difficulty lies in (a) the predecessor not fully understanding itself and its goals, and (b) the successor needing to act with some degree of autonomy. (Text versions only.)
    • Put more explicit focus on the case where a successor is much smarter than its predecessor. (Text versions only.)
    • Expanded "Usually, we think about this from the point of view of the human." to "A lot of current work on robust delegation comes from the goal of aligning AI systems with what humans want. So usually, we think about this from the point of view of the human." (Text versions only.)
  • Goodhart's Law:
    • Expanded discussion of regressional Goodhart, including adding more illustrations and noting two problems with Bayesian estimators (intractability, and realizability). Removed claim that Bayes estimators are "the end of the story" for regressional Goodhart.
    • Moved extremal to come after regressional instead of after causal, so extremal and regressional can readily be compared.
    • Rewrote and expanded extremal Goodhart to introduce the problem more slowly, and walk through quantilizers in much more detail.
    • Expanded discussion of causal Goodhart to clarify connection to decision theory and note realizability issues.
    • Clarified the connection to mesa-optimizers and subsystem alignment in adversarial Goodhart.
  • Stable Pointers to Value:
    • Added following the Goodhart discussion: "Remember that none of these problems would come up if a system were optimizing what we wanted directly, rather than optimizing a proxy."
    • Introduced the term "treacherous turns".
    • Shortened and clarified introduction to observation-utility maximizers, described how observation-utility agents could do value learning, and removed mention of CIRL in this context.
    • Mentioned the operator modeling problem.
    • Discussed wireheading as a form of Goodharting.

Changes to "Subsystem Alignment":

  • "Optimization daemons" / "inner optimizers" are now "mesa-optimizers", matching the terminology in "Risks from Learned Optimization". (Change also made in "Embedded Agents" / the introduction.)
  • New section on treacherous turns, simulated deployments, and time and length limits on programs.


Discuss

Objective Dog Ratings: The Irish Wolfhound

Новости LessWrong.com - 29 августа, 2020 - 07:01
Published on August 29, 2020 4:01 AM GMT

Like other sighthounds, Irish wolfhounds are thin, wiry, and not a little bit tall. Indeed, wolfhounds are the tallest of the Tall Dogs. In personality, I have no complaints: they are intelligent, respond well to training, and get along with people so well that they often make terrible guard dogs. They are very unlikely to eat a small child, and that is a definite improvement over the Original Wolf™. 

Unfortunately, wolfhounds look unkempt and scraggly, like they just got out of a week-long bender. This is not a dog which cares about maintaining a professional appearance. They mostly die of bone cancer, which is less the fault of bad genes than of size (other large dogs are also prone to bone cancer), but they also have a predisposition to heart problems which does seem genetic in nature. 

Irish wolfhounds are perfectly adequate dogs, but they need to dress for the job they want, not the job they have, and their numerous genetic bottlenecks (with subsequent inbreeding) are also a cause for concern.

Rating: ★★☆ (Fine)

Original post (w/ more pictures) here.



Discuss

Objective Dog Ratings: An Introduction & Explanation

Новости LessWrong.com - 29 августа, 2020 - 06:54
Published on August 29, 2020 3:54 AM GMT

Because they’re not all good dogs, Brent.

Obviously, not everyone wants the same thing from dogs, but that’s not necessarily the same as saying that any dog is as good as the next, or even that any dog rating system must be subjective. If we’re going to entertain the idea that some dogs are simply better than other dogs, though, then we have to specify how that is.

What we’re looking for is a dog with a certain amount of wolfishness, a dog which is as close to being a wolf as one can get without sacrificing any of those essential characteristics which define a dog as such. Basically, a dog which a politically progressive, forward-thinking wolf would not be ashamed to know. They must be loyal, intelligent, and hardworking, they must have a sense of dignity, they must like humans, and above all they must be healthy. A dog which is perfect in every other way, but is unhealthy, is a bad dog, because it would not be good to be that dog. 

Unfortunately, I’m incapable of doing anything without taking it at least a little bit seriously, so ratings will be on a three-star scale, from ★ (Mediocre) to ★★★ (Good). It is also possible that some dogs will not get any stars at all. Those are bad dogs, Bront. 

I do not know which dog breed will turn out to be the dog breed, the dog of the gods, but I do have my suspicions (some breed of spitz-type, probably), and it’s important to note that this has nothing to do with how much I personally like a dog. Some of my favorite dogs will get no more than two stars, and some may even get just one star. This isn’t “Dogs which are the best at sitting in my lap and being petted,” or “Most Instagrammable dogs.” 

Let us begin.



Discuss

The Case for Human Genetic Engineering

Новости LessWrong.com - 29 августа, 2020 - 04:33
Published on August 28, 2020 10:21 PM GMT

This is the first post in what I hope will be a series of posts arguing that genetically engineering humans may provide a huge benefit to individuals and society as a whole. In the interest of creating something readable, this post will largely ignore the controversies and unintended consequences of such a project, but I plan to address those in later posts. It will also ignore perhaps the most impactful genetic change of all: increased intelligence. Such a change deserves a post of its own.

A quick disclaimer before I begin: when I first sat down to write a post about genetic engineering, I planned to thoroughly research everything I wrote about and give links to most of the claims I made. While I will do so for what I judge to be the less commonly understood facts presented in this piece, this will not be as thoroughly researched and comprehensive as I originally planned.

As a result, many of the conclusions I draw in this piece will be based on my own incomplete knowledge and are therefore liable to be wrong. If you spot any particularly glaring errors, or if the pacing is off, or if you get too bored and don't finish reading please let me know in the comments. That being said I think I have read enough about this topic to have something worth reading.

Part 1: A Changing World

Human history is a story of accelerating change. The rapid growth in brain size and general intelligence that took place between 3 million and 50,000 years ago enabled the explosion of human populations and power that culminated in our modern globe-spanning civilization. There is still some debate in the field of anthropology about WHY exactly evolution favored larger brain sizes and increased intelligence so consistently for so long. Whatever the reasons were, they must have been very compelling. Relative to resting metabolic rate -- the total amount of calories an animal burns each day just to keep breathing, digesting and staying warm -- the human brain demands more than twice as many calories as the chimpanzee brain, and at least three to five times more calories than the brains of squirrels, mice and rabbits.

This massively increased brainpower had one particularly notable effect: humans became able to communicate via language, a far more flexible and sophisticated form of communication than that used by any other species. This unique ability probably played a fundamental role in the development of agricultural societies, which was the first step in the march towards modern civilization.

The agricultural revolution led to an explosion in the size of the human population, and the industrial and green revolutions lead to a rate of population growth unprecedented in human history. This massive population growth and increased technological sophistication has dramatically altered human lifestyles. For most of human history, individuals lived in groups of at most a few hundred and subsisted off of a combination of hunting, gathering, fishing, and scavenging. This lifestyle gave us many of our current traits including our upright posture, our teeth (which are optimized for eating a combination of meat and plants), our large brain sizes, our penchant for gossip, and many other human characteristics.

When agriculture spread throughout the world beginning around 12,000 years ago at the end of the last ice age, it dramatically altered human lifestyles and diets. Humans began to live shorter less healthy lives, back neck and tooth problems became much more prevalent and diseases began to spread in the dense sedentary societies that sprung up around the world (particularly in Asia and Europe).

In a very real sense, the agricultural revolution made life worse for the average human. But because life was not so bad that sedentary individuals were less likely to pass on their genes, and because agriculture could support far more humans with the same land area, there was no path back. Humans across the planet turned to agriculture not because it provided for a better, happier life, but because they were stuck in a Malthusian trap.

A Genetic Mismatch

The decline in lifespan, decrease in height, increased incidence of bone and joint issues, the rise of cavities, and the spread of infectious diseases that accompanied the agricultural revolution are attributable to a mismatch between human genes and human lifestyles. It is my contention that despite significant improvements in lifespan, sanitation, and food supply, the rapid progress of modern technology is creating a wider and wider gulf between the environment humans evolved to live in and the one in which we find ourselves today.

Humans are quite adaptable, so we have created ways to bridge the gap between these biological needs and the shape of modern living. Gyms and exercise equipment, for example, give people a way to maintain their physical and mental health in the absence of lifestyles that necessitate exercise as a required part of staying alive. But these solutions are extremely sub-optimal: humans now have to spend several hours per week running, swimming, biking and lifting weights for no particular reason other than to maintain health. And while many people might argue that “exercising makes me feel better and look better and live longer” (all true by the way), it is still the case that our ancestors got the same benefits in the process of doing something they had to do anyways (hunting and gathering).

There are many, many other such examples. Tooth issues such as wisdom teeth crowding out other teeth in our jaw, the frequency of cavities and tooth decay are also an example of a problem introduced by a change in our diet that accompanied the agricultural revolution. Frequent back and neck issues are also a result of a mismatch between our ancestral environment and our modern working conditions. Our tendency to focus on gossip about the lives of celebrities whose lives will never impact us is a relic of an ancestral environment in which the only people whose gossip we heard were those in our tribe (about whom it was useful to know gossip). Our preference for sugary foods devoid of essential nutrients are a relic of an era in which such foods were hard to come by and the risk of starvation was a much greater risk to reproductive success than the risk of obesity. And the disproportionate attention we pay to extremely low probability risks like terrorism and violent crime are a relic of an era in which human to human violence was much more common than it is today.

The incredibly high frequency of death from old age represents perhaps the greatest disconnect between the environment our genes were optimized for and the one in which we now live. As explained in this excellent quora post by Dr. Suzanne Sadedin, the average age at which an individual organism from a given species will die is determined by the rate of all-cause mortality in its natural environment. This evolutionary theory of aging, known as the Antagonistic Pleiotropy Hypothesis, is well supported by theoretical models, animal experiments and human correlational studies. The mechanism of action here is a set of genes with a specific characteristic: they increase reproductive fitness at a young age but decrease the window of reproductive opportunity (often by causing health problems at an older age). When all-cause mortality is high, such genes are beneficial as the organism carrying them is likely to have died by the time the downsides become relevant.

So if the antagonistic pleiotropy hypothesis is to be believed, how long would we expect humans to live for if they were genetically optimized for their current environment? Unfortunately, I wasn’t able to find any models predicting lifespan given all-cause mortality rates of a particular species. However, let us compare the mortality rates of hunter-gatherer societies with those of humans living in the developed world to give us a sense of how massive the difference is. Here’s a graph showing mortality rates in various Hiwi hunter-gatherer groups.

Here's another graph showing mortality rates in Canada.

It isn’t even close. The chance of death between the ages of 1 and 5 are somewhere between ten to thirty times lower in modern societies than in hunter-gatherer societies, and even at age 70 mortality rates are still at about a third of the levels they are in hunter-gatherer societies.

It therefore stands to reason that we could substantially increase the human lifespan by opting for genetic variants that give slightly lower reproductive fitness at a young age in exchange for longer life. It also stands to reason that given the low rate of all-cause mortality in modern society, this trade-off would INCREASE reproductive fitness.

There are many more things that were clearly important considerations in the past that are not as important today. For example, the cost of gaining access to more calories is not as high today as it was in the past. Are there genes that increase health or intelligence at the cost of increasing one’s basal metabolic rate? If so, such genes might have been selected against in the past. But with much easier access to calories today, such genes might provide a net benefit. Are there genes that increase intelligence at the cost of a larger fetal skull size? Babies with such genes might not have fit through the birth canal in the past, but we now perform c-sections on a regular basis. The possibilities here seem absolutely enormous and we already have specific examples of genes with trade-offs that don’t make sense anymore. Are there genes that increase the frequency and severity of the stress response, making us better at fighting off predators and other humans at the cost of longevity? If so, perhaps we decrease the expression of such genes to increase lifespan at the cost of not being able to win bar fights or do amazingly well at contact sports. You get the idea.

Part 2: Surpassing Evolution

Evolution works wonders over long timescales, but it is not efficient or even good at maximizing reproductive fitness. As Eliezer Yudkowsky once wrote, “the wonder of evolution is not how well it works, but that it works at all.” Such a process leaves much to be desired. In this section, I will be describing how genetic engineering will allow us to surpass the fitness maximizing constraints imposed by evolution, and by doing so improve the lives of humans and the rest of this planet’s species.

The first limitation I will be discussing is that of the local fitness maxima. One of the most frustrating things about evolution is that it can only make progress one mutation at a time. If gene B only provides a benefit when gene A is already present, gene A must spread through a breeding population before gene B. And if gene A does not by itself provide a reproductive fitness advantage, it becomes nearly impossible for gene B to ever spread. There are some exceptions to this (see Scott Alexander's excellent post on how weak competition can actually lead to increased fitness), but in general, this is the rule.

Genetic engineering opens up the possibility of escaping from the “local fitness maxima” created by this one-step-at-a-time limitation of evolution. I’m going to tell you the story of one of the most promising such interventions I know of: the project to move genes out of the mitochondria and into the nucleus of cells.

MitoSENS: Lending Evolution A Hand

MitoSENS is an ongoing project to address one of the fundamental causes of aging: damage to mitochondrial DNA caused by free radicals.

This story begins 1.45 billion years ago, when, during an unbelievably rare occurrence, a large cell swallowed a small one, the small one survived and multiplied inside the larger one and neither one died. This small cell was special: it was the ancestor of modern mitochondria, and it dramatically increased the amount of energy available to the large cell. This event was a seminal moment in evolutionary history, surpassed in significance perhaps only by the origin of life itself. As best we can tell, it only happened a single time in the 3.5 billion year history of life, and from that single ancestor all eukaryotic organisms (plants and animals) are descended.

For this reason, mitochondria (along with chloroplasts) are the only organelle in eukaryotic cells that can self-reproduce. A legacy of this independent origin story lives on within the membrane of every mitochondrion: 37 genes and 16,569 base pairs which form the last remaining vestiges of an organism that once lived independently in a much larger world.

You might suspect that 37 genes are not nearly enough for any organism to function, let alone reproduce. You would be correct. This was a bit of a mystery to me as well until I learned what evolution has been doing to mitochondrial DNA over the last billion years of evolution: it has been moving DNA out of the mitochondria and into the nucleus.

You see, mitochondria are one of the single biggest sources of free radicals in our bodies. In fact, the free radicals (AKA reactive oxygen species) that are produced by our mitochondria account for the vast majority of free radical damage in an average person’s body. The inside of a mitochondrion is one of the worst places to be if you are a molecule that values your current atomic arrangement. With no nuclear membrane to protect itself, mitochondrial DNA is exposed to the full fury of this onslaught of free radicals produced as a byproduct of ATP synthesis.

So the process of random mutation and natural selection has been hard at work moving genes out of the mitochondria and into the nucleus of the cell. I still haven't found a satisfying explanation of exactly HOW this transfer happens, but some process appears to have been hard at work over the last 1.5 billion years moving genes out of the mitochondria and into the nucleus of the cell. Proteins necessary for mitochondrial function and now produced outside the mitochondria and transported back inside via the TIM-TOM complex, a series of channels in the membranes of each mitochondrion that allow externally manufactured proteins to be moved inside the mitochondrion. This evolutionary process has moved almost all of the 3000 genes of the ancestor of mitochondria into the cell's nucleus. But evolution can only advance one step at a time, and there’s something special about those remaining 37 genes that makes them particularly resistant to evolution’s effort.

Two chief problems appear to be at the root of evolution’s inability to move those remaining genes out of the mitochondria: hydrophobicity and code disparity. Code disparity is a difference in the interpretations of codons in the nucleus and the mitochondria. A codon is a set of 3 base pairs that represent an amino acid or a regulatory signal such as "end of protein". At some point in evolutionary history, the interpretation of four of these codons was switched in the mitochondria. The first of the four that appears to have changed its interpretation is the codon formed by the base pairs UGA. UGA is used to encode a STOP signal (meaning the end of a protein sequence) in nuclear DNA. But some time around 1 billion years ago this codon’s interpretation was switched from being a STOP signal to encoding the amino acid tryptophan in the mitochondria. Once this happened, gene transfer from the mitochondria to the nucleus became significantly harder, because the proteins synthesized from such genes would be truncated at the location of every tryptophan in the structure.

The rest of the paper explaining why no more genes seem to have transferred is quite interesting and can be read here if you’re interested.

This is of importance because mitochondria free radical damage appears to play a critical role in aging via a process called the "Mitochondrial Free Radical Theory of Aging".

A full explanation of the theory is beyond the scope of this post (read chapter 5 page 68 of Aubrey de Grey's book Ending Aging if you want one.) But the shortest version ever is that a small proportion of mitochondria accumulate a specific set of mutations with age that turns the cell in which they reside into toxic waste production facilities. The ATP synthesis process that Mitochondria normally perform is shut down inside such cells, forcing them to turn to another energy production method whose byproduct is superoxide, a dangerous free radical. These free radicals end up colliding with low-density lipoprotein and creating oxidized cholesterol, one of the primary contributors to high blood pressure and heart disease.

I should point out here that the following explanation is not universally accepted. There is at least some criticism of the “Mitochondrial free radical theory of aging” proposed by de Grey, and the issue doesn’t seem quite settled one way or the other. However, given evolution’s long history of moving mitochondrial genes into the nucleus, it seems very likely that there is a fitness advantage to doing so even if a reduction in the rate of aging is not THE specific reason.

Since we know how to translate mitochondrial genes into nucleus-encoded genes by swapping the codons that cause the code disparity, we could engineer nuclear copies of all the genes. Even after the genes inside the mitochondria are damaged, imported proteins would allow the mitochondria to continue functioning, preventing not only a significant portion of aging damage but simultaneously providing a cure for several dozen mitochondrial genetic diseases such as Leber Hereditary Optic Neuropathy (LHON) and Kearns Sayre syndrome. In fact, clinical trials to express the protein that causes LHON in the nucleus are in clinical trials right now

In short, genetic engineering might allow us to permanently fix a significant source of aging damage and genetic disease with no significant downsides.

Promoting Heterozygous Advantage

Sickle cell anemia is an interesting genetic disease. It is caused by a mutation in the gene that codes for the protein hemoglobin, which is responsible for carrying oxygen in the blood. The disease is recessive, meaning only an individual with two copies of the gene will experience disease symptoms. Those suffering from the condition are often wracked with pain, have restricted blood flow to vital organs, and have difficulty performing moderate exercise.

Carriers (people with one normal copy of the gene and one mutated copy) have an interesting advantage not enjoyed by the rest of us: they are notably more resistant to malaria. Other than this, they only seem to have symptoms under extreme dehydration or oxygen deprivation.

Carriers of the sickle cell disease, therefore, have a notable fitness advantage in environments in which a low percentage of the group of available partners are carriers and the risk of death or disability from malaria is high. This is why when we look at maps of the distribution of malaria and the distribution of people who have (or whose ancestors had) sickle cell, they overlap quite nicely.

Ancestral homeland of individuals with sickle cell anemia

Historical range of malaria

Genetic engineering offers us the opportunity to avoid the “overdominance” problem of genetic conditions like sickle cell: we can ensure that EVERYONE in areas where malaria is a major risk has exactly one copy of the sickle cell gene. In other words, we can reach population states that evolution simply cannot.

Avoiding Losses from Zero-Sum Games

I left this example for last because I do not yet have a specific example of this phenomenon in humans, though I suspect that some exist.

Walk into any forest of old trees and you will likely notice that the first hundred feet or so of the trunk are devoid of any branches. In the competition for access to sunlight, trees grow nearly as tall as physiologically possible in an effort to pass the shading branches of their neighbors. While this tendency is a huge boon for lumber companies that take advantage of the long straight trunks to create lumber products at low cost, the trees themselves do not on net benefit from the arrangement. Each tree must invest considerable energy in producing a hundred or more feet of wood whose sole purpose is to elevate its canopy above those of its neighbors.

The forest as a whole is less successful than if all trees were to grow tall enough to spread their canopies fully but no taller. But alas, the trees have no mechanism for punishing uppity young saplings that dare to grow taller than their older neighbors. So all trees are forced to grow tall and the reproductive fitness of the forest as a whole is reduced.

This is a fairly standard example of the prisoner’s dilemma, a phenomenon in which two self-interested entities compete in a game, and both end up losing due to the lack of ability to punish cheaters. If you are not already familiar with the concept I would highly recommend reading the link above as it does a much better job explaining the setup than my one-sentence summary.

Though I don't have any specific examples, there likely exist specific genetic variants that impose a cost and exist solely to allow humans to compete better in zero-sum games. If we are able to identify such variants, it's possible that we could ban humans from having such variants, thus saving everyone from the cost of carrying such traits. Obviously such a scheme would carry some risk and may be rejected by most people as giving the government too much power, but it is nonetheless a benefit that can only be realized through genetic engineering. For that reason, it

Future Posts

I hope to continue this series. I'd like to devote an entire post to the topic of genetically engineering higher intelligence since this would likely be one of the most important things that we would choose to change. I'd also like to discuss HOW this could actually be done via embryo selection, gene-editing tools like CRISPR, and iterated embryo selection.

Let me know what you thought of this post. My goal here is really to create something that's informative and readable. So if this post could use improvement in either of those areas please let me know.



Discuss

My guide to lifelogging

Новости LessWrong.com - 29 августа, 2020 - 00:34
Published on August 28, 2020 9:34 PM GMT

I've defended the practice of lifelogging as a means of life extension here. In this post I'll provide a fairly comprehensive guide on how to lifelog. Since lifelogging exists on a spectrum from "taking a picture every so often" to "recording every single detail of your life, in uncompressed HD video along with continuous MRI scans and storage in a nuclear-safe vault" this guide will present two categories for lifelogging, the first for lower cost options and the second higher cost options. "Cost" here refers not only to the monetary price of buying the equipment, but also the convenience costs of setting up the equipment, and storing the data, and perhaps social embarrassment.

Over the last several months I have spent many hours of research to determine the best setups in terms of time and energy required to record my life. I also recommend viewing Mati Roy's setup.

I intend to update this guide as I learn more, so keep in mind that this post is a work-in-progress.

Lower cost lifeloggingArchiving social media

The lowest hanging fruit of lifelogging is probably creating a long-term archive of your social media data. The method of archiving your social media data will necessarily depend on the websites you visit, but here are some guides for common websites:

Keylogging

I do not keylog myself, but Mati Roy has informed me that Spyrix works well.

Taking pictures

These days, smartphones generally have high quality cameras, and are much less of a hassle than buying professional equipment.

Storage

Since everything in the lower cost section here takes up a small amount of space, cloud storage is an appropriate way of storing data for the long-term. Google Drive offers 15 GB of free storage, though I would also suggest storing a local copy of every file along with checking out the subheader on audio and video compression, and the subheader on long-term storage in the section on higher cost lifelogging.

Higher cost lifeloggingScreen recording

The most salient way that I lifelog is by recording everything that happens on my computer screen, along with a full video of my face and room. I achieve this setup by using Open Broadcaster Software (OBS) and record a continuous split-screen between my screen and my USB camera.

I have heard that OBS can be quite annoying to use if you use a laptop, as it turns up the fans and generally uses too much of the CPU. Therefore, I recommend building a computer with a high quality CPU and buying a wide-angle USB camera, along with a USB microphone to record.

The desktop computer I use is five years old, so I cannot recommend the exact parts I bought at the time. I also do not recommend using the USB camera that I bought, as it does not have a wide enough angle for my tastes. Instead, I recommend browsing the subreddits /r/buildapc and /r/buildapcforme until you have a decent idea of what goes into building a computer.

I would estimate the minimum cost of a desktop computer that can reliably run OBS without problems at around $500, if you know what you are doing. But a price tag of $750 may be better if you don't want to run into issues later. This benchmarking site, and this one are useful for determining low cost high quality CPUs. At the moment I suggest getting around a $200 to $300 newest generation AMD Ryzen CPU.

Nighttime recording

The value of recording yourself sleep is arguable, so I do not suggest this to everyone. My own justification was to have a sense of completeness in my lifelogging, and feel like I wasn't ever missing a moment.

That said, I purchased this USB infrared camera to record myself at night, and it works well. It also functions as a day-time camera, automatically switching to infrared when the lights go out, making it suitable as an all-day recording camera. I also purchased this fitness watch to track my sleep, though this aspect is obviously not necessary.

Just as in the above section, I use OBS to facilitate the recording. It's worth understanding how profiles and scene collections work in OBS so that you can simplify your setup.

When I'm not at my desktop

Recording at my desktop is nice, since I can use OBS, but when I'm on-the-go I have two main ways of recording, using audio and video.

Audio

The first method is audio recording using my phone. I have an iPhone at the moment, and therefore I recommend Android users to look at Mati Roy's advice. I purchased this omnidirectional lavalier microphone along with this lightning-to-headphone jack connector, and am generally pleased with the quality.

I use the app Dictaphone, but I'm not confident at all that this app is the best. It was simply the first thing I looked at for IOS.

The lavalier microphone connects to my shirt, sort of like in this picture, and I don't generally have to think about it much when I'm on the go. Of course, I urge potential lifeloggers to make sure that they have the consent of all parties before recording people on-the-go.

In order to save phone battery, I also purchased this voice recorder, which has surprisingly long battery life and acceptable storage. However, I mostly don't use the voice recorder anymore since I have switched to mainly recording video while I'm on-the-go, as I explain in the next section.

Video

If you aren't satisfied with recording audio continuously on-the-go, you can switch to using video. I experimented with purchasing an action camera (ie. what Go Pros are) for this purpose, but then soon realized that there was a better alternative.

I now recommend lifeloggers purchase a body camera, of the type used by police. Here some of the pros and cons of body cams compared to action cameras:

Pros:

  • They generally have much longer battery life (very important)
  • Most have native infrared recording so you can record at night
  • Body cams are built to allow you to easily clip it onto clothing (this makes continuous recording less awkward)
  • They tolerate shock damage, such as dropping the camera, more than many action cameras

Cons:

  • The video quality is lower
  • Fewer features are available
  • There are fewer online resources for operating body cams

After substantial research I decided to buy this body camera. The primary reason I went with it over other cameras was because it had a detachable battery (with an extra), and detachable storage (but you must purchase the SD card on your own). The main downside is that the lens angle is only 140 degrees compared to 170 in some other body cams.

The body cam is well-built and is much lighter in weight than you might expect. It connects easily to my computer via a USB cable that enables me to transfer the video files to long-term storage.

To minimize storage costs, I record in 480p and compress all my files once I have transferred them to my computer (see next section). The body cam allows an option for on-board storage during recording, but I don't use it because it seems to work by simply halfing the bitrate of the video without anything intelligent involved. A similar thing seems to happens when you turn on the option for pre-recording for some reason.

On a full charge I can get over 5 hours for each battery, and it is easy to replace the battery when the body cam dies. With a 128 GB SD card it can hold about 60 hours of continuous 480p quality video before it runs out of space.

I have tried various ways of connecting it to my body, and the thing that seems to work the best is simply connecting the body cam to my pants, or belt, as shown in the image below. Unfortunately, you do have to tuck in your shirt or else the body cam won't be able to see much. On the bright side, this means that if you want to hide that you are wearing a body cam, all you have to do is make sure your shirt covers it.

More downsides of keeping it on your pants include the fact that it doesn’t get people’s faces if you are talking to them and you are close to them, and it doesn't record very well when you are sitting down at a table or sitting more generally.

See this Google Drive video for a sample of the post-compression quality of using the body cam. I think that Google Drive compresses video uploaded there, so make sure to download it to see the real quality, as it's only 21.9 MB.

Audio and video compression

Video takes up a LOT of storage unless you compress. Audio is similar, though less extreme. Therefore, before transferring my files into long-term storage, I always compress them into something of acceptable size.

I use FFmpeg to compress my media files, which works well on Ubuntu, but I have not tried it out on other operating systems. To compress my videos I run this bash script,

#!/usr/bin/env bash

for i in *.MOV;
do name=`echo "$i" | cut -d'.' -f1`
echo "$name"
ffmpeg -i "$i" -c:v libx264 -preset veryslow -crf 24 -strict -2 "${name}.mp4"
done

rm *.MOV

The two most important things to understand about the script above are the options

-preset veryslow

and

-crf 24

These options determine the quality and size of the video. I recommend choosing quality and size depending on your own tolerance for storage costs (see the section on long-term storage below). The the FFmpeg documentation explains these options in more detail,

A preset is a collection of options that will provide a certain encoding speed to compression ratio. A slower preset will provide better compression (compression is quality per filesize). This means that, for example, if you target a certain file size or constant bit rate, you will achieve better quality with a slower preset. Similarly, for constant quality encoding, you will simply save bitrate by choosing a slower preset.Use the slowest preset that you have patience for. The available presets in descending order of speed are ultrafast, superfast, veryfast, faster, fast, medium – default preset, slow, slower, veryslow.[...]The range of the CRF scale is 0–51, where 0 is lossless, 23 is the default, and 51 is worst quality possible. A lower value generally leads to higher quality, and a subjectively sane range is 17–28. Consider 17 or 18 to be visually lossless or nearly so; it should look the same or nearly the same as the input but it isn't technically lossless.The range is exponential, so increasing the CRF value +6 results in roughly half the bitrate / file size, while -6 leads to roughly twice the bitrate.

For audio, I use this command,

find -name "*.WAV" -exec ffmpeg -i {} -acodec libmp3lame -qscale:a 5 -ab 128k {}.mp3 \;Long-term storage

For large amounts of short-term storage, you can visit the website diskprices.com to view the cheapest storage available to consumers. Personally, I recommend getting SSD storage as opposed to HDD storage for short-term use, as even though it is more expensive, it is also much faster.

However, since both SSDs and HDDs are not built to store data for decades without corruption, the best option at the moment is likely burning data onto blu ray discs. You can find cases of 50 blu ray discs that hold 22.5 GB for between $20 to $25. However, the real costs of long-term storage will be higher than this for two reasons,

  • For the long-term, ideally you should keep at least two copies of every file, and you should store them in separate locations.
  • Data burning often fails with a rate of between 10 to 20 percent, which means that your true cost estimates should take into account the fact that many discs will be useless.

I purchased this blu-ray burner, which works acceptably but I'm unsure whether it is the best option. I also purchased a few of these cases which can hold a lot of discs quite cheaply.

I suggest taking a look at a list of best practices for long-term storage on blu-ray discs as compiled by Brian Tomasik. Like him, I am not an expert either, so take this advice with a grain of salt.



Discuss

How to teach things well

Новости LessWrong.com - 28 августа, 2020 - 19:44
Published on August 28, 2020 4:44 PM GMT

(This is a post on my thoughts on good teaching techniques from a daily blogging project, that I thought might be of interest to LessWrong readers)

Introduction

This is a blog post on how to teach things well. I’ll mostly be focusing on forms of teaching that involve preparation and structure, like talks and tutoring, but these ideas transfer pretty broadly. I think teaching and explaining ideas is an incredibly important skill, and one that most people aren’t great at. I’ve spent a lot of time practicing teaching ideas, and I think I’ve found a bunch of important ideas and approaches that work well. I’m giving a talk next week, so I’ll initially focus on how to give good talks, but try to outline the underlying concepts and high-level ideas of teaching. And then talk about how these can transfer to contexts like tutoring, and to teaching specifically maths or applied rationality - the main areas I have actual teaching experience with.

Note: I mostly care about teaching concepts and ideas, and teaching things to people who genuinely want to learn and be there, so my advice will focus accordingly.

I think it’s useful to think about good teaching even if you don’t intend to spend much time teaching - learning and teaching are flip sides of the same process. I’ve found that even when in the role of a student, understanding what good teaching looks like can often fix a lot of the shortcomings of a bad teacher!

Framing

The key insight of this post is that good teaching requires you to be deliberate, and keep the purpose in mind: learning is a process of information compression. When you’re learning something new, you essentially receive a stream of new information. But human cognition doesn’t work by just storing a flood of information as is. The student takes in the information stream, extracts out the important ideas, converts it to concepts, and stores those in their mind. This is a key distinction, because it shows that the job of a teacher is not to give the student information, it’s to get the student to understand the right concepts. Conveying information is only useful as a means to an end to this goal.

In practice, it often works to just give a stream of information! Good students have learned the skill of taking streams of information and converting it to concepts. Often this happens implicitly, they student will absorb and memorise a lot of data, and over time this forms into concepts and ideas in their head automatically. But this is a major amount of cognitive labour. And a good teacher will try to do as much it as possible, to let the student focus their cognitive labour on the important things.

My underlying model here is that we all have a web of concepts in our minds, our knowledge graph. The collection of all the concepts we understand, all of our existing knowledge and intuitions, connected together. And you have learned something when you can convert it to concepts and connect it to your existing understanding. This means not just understanding the concept itself, but understanding where it fits into the bigger picture, where to use it, etc.

The final part there is key - if the student leaves with a good understanding of the ideas in the abstract, but no idea when to think about the ideas again, it’s no better than if they’d learned nothing at all. We call on our knowledge when something related triggers, so in order for a lesson to be useful, you need to build those connections and triggers in the student’s mind.

A key distinction to bear in mind is ideas being legible vs tacit. A legible idea is something concrete that can easily be put into unambiguous words, eg how to do integration by parts. While tacit knowledge is something fuzzier and intuitive, eg recognising the kinds of integrals where you’d use integration by parts in the first place - essentially the intuitions you want the student to have. This is a good distinction to bear in mind, because legible knowledge is much easier to convey, but often your goal is to convey tacit knowledge (at least, it should be!). And there’s a lot of skill to conveying tacit knowledge well, and making it as legible as possible without losing key nuance. And different techniques work better for the two kinds. A lot of my issues with the Cambridge maths course is an extreme focus on legible knowledge over tacit - the underlying intuitions and motivations.

How to teach

There are two key problems when teaching, that any good teaching advice must account for:

  • Limited ways to convey information
    • The ideas I’m teaching are stored as implicit concepts in my mind, but in order to convey them, I must translate them into language. This language maps to ideas in my head according to all of my implicit knowledge and worldview, but translates into the student’s head according to their implicit knowledge and worldview. This often creates errors
    • And converting concepts to language is inherently lossy, ideas have a lot of tacit nuance that is hard to capture
    • Essentially, words can only convey legible knowledge, and I need to figure out how to hack this to convey tacit knowledge. Or how to find alternate information channels
    • Alternately, I need to have error checking mechanisms to notice when I’ve failed to convey something well.
  • Typical Mind Fallacy
    • A key part of learning is combining knowledge you hear with the ideas already in your head.
    • But I only have access to what’s in my head, not what’s in the student’s. And by definition this is different - I already understand what I’m teaching!
    • This is a crippling blow to my ability to teach, and I need to be constantly aware of this and trying to build models of what’s in the student’s heads, and how they receive what I’m saying.

Here are some of the most important tools I have for addressing these problems:

  • High-level picture
    • The student’s knowledge graph is big. So the first, and most important part, is identifying which part of the graph they should add these ideas to
    • Thus you should always highlight where this fits in to the bigger picture. Which questions are we currently trying to answer? Why is this idea interesting at all? Where can the student use this? What are its limitations?
    • This should always be the first thing when introducing a new idea
  • Prioritising information
    • Learning is information compression. This fundamentally means that the student needs to be paying selective attention. When learning something new, the Pareto Principle always applies - 80% of the importance lies in 20% of the ideas.
    • Identifying this 20% is significant cognitive labour, because it’s not immediately obvious. They need to pay attention to everything and later filter.
    • But, the question of “what matters here” is tacit knowledge that I already have! This talk is high labour for the student, but easy for me. Thus, the most important thing a teacher can possibly do is to highlight what matters and what doesn’t, to tell the student where to focus their attention.
    • In practice, you should always be saying “this is really, really important” or “these are just fiddly details” or “this is a bit of a niche edge case” etc. It is extremely hard to do this too often.
      • And give more time to the important things. If you say an important point, write it down and put a box around it. Explain why it’s important and how it fits into the bigger picture. Give an alternate explanation, or an example.
      • This is really easy to miss if you think of teaching as information transfer - where your goal is to tell the student everything and hope they figure out what to pay attention to themselves.
    • Another tool: Have frequent summaries, highlighting the key points in the previous section.
      • This is further useful to highlight connections. Some ideas will fit into their knowledge graph more easily than others, and pointing out connections can leverage the easy connections to make hard ones easier
    • I’m a big of the advice “say what you’re going to tell them, tell them, and say what you’ve just told them”, I think it’s a good way to implement this principle
    • The fundamental principle behind this is that students retain a tiny fraction of what they hear. If you give a one hour talk, they’ll retain maybe a few minutes worth of content. This is a fundamental fact of the learning process, and the only thing you can do about it is to control what they retain. Focus their attention on the important parts
      • This mindset helps me identify what’s important. Set a 5 minute timer, and write down everything important that you want people to retain. These are the key points around which your talk should be structured!
      • Everything else you say should be intended to help these key points stick better - to highlight connections between them and to existing knowledge, to ensure good information transfer, and to convey the tacit knowledge underlying the key points
    • Another key skill of prioritisation is cutting things. If one part is irrelevant, or it’s boring and fiddly, cut it! It’s painful to not talk about everything cool, but you have limited time - if you don’t actively prioritise, you aren’t avoiding trade-offs, you’re just ceding control to “whatever you leave last”
      • Anyone who’s gone to a talk by me knows that I have yet to internalise this lesson
    • If there’s one point you retain from this post, let it be this one - this is incredibly important, and the main mark of a good teacher vs a mediocre one
  • Understand pre-requisites
    • A consequence of the Typical Mind Fallacy, is that it’s super easy to forget that your students don’t have all the context you have! This manifests as people teaching ideas without the pre-requisites.
      • The underlying idea: the ideas you teach are in your knowledge graph, and are built upon existing ideas - these are the pre-requisites. You need to figure out whether the students
    • A common secondary mistake is to understand pre-requisites, and then try to explain all of them!
      • A good framing here is inferential distance. The inferential distance of a new idea is the number of steps of new concepts someone must understand before they can understand the new idea. Eg, to teach a young kid about the quadratic formula, they first need to understand the idea of polynomials, for which they need to understand algebra - this is three inferential steps.
      • A general rule of thumb: never teach things with more than 2 inferential steps. An idea just learned is shaky, and doesn’t yet have good connections built. It’s very, very hard to anchor new ideas onto new ideas.
    • This is really hard to get right. Pre-requisites require you to have a good model of somebody else’s mind. A useful technique is often to do a practice run on someone from your target demographic, and ask them to flag everything that confuses them.
    • Further, for groups, this can be an intractable problem, everyone has different prior knowledge. You need to have a clear picture in your head of who the talk is aimed at.
  • Students should learn actively, not passively
    • It’s really easy for a student to just passively sit in a stream of information, taking none of it in. This achieves neither of your goals, because to form connections, compress information and connect it to their knowledge graph, they need to be putting in some cognitive labour.
    • Good ways to encourage this: explicitly telling them that this is important, giving exercises and time to think through them, asking the audience regular questions and giving them some time to think.
      • Question asking has the failure mode where most people won’t volunteer, or just zone out a bit - I lack great solutions to this problem
    • This is much easier to handle in smaller settings, I’m bad at handling it in talks. The main solution I have is just to be engaging and keep the pace going well.
      • Often people will zone out, so having regular breaks, and check-points of “if you weren’t following, we’re changing topic so it doesn’t matter” can help to rectify this
        • Even short, 30s-120s breaks can be helpful! Encourage people to get up and stretch.
        • Breaks never feel important, but they really, really help
  • Give examples
    • Examples are an insanely powerful tool for teaching things well, and people rarely use them enough. I have never given a class with too many examples (and believe me, I’ve tried)
      • “You can teach a class with no content, only examples; you can’t teach a class with only content, no examples”
    • Why examples are awesome:
      • Examples are an excellent way to resolve lossy information transfer - they’re a completely different channel of communication than normal. If nothing else, they serve as an error check
      • Examples are a great way to transfer tacit knowledge, without necessarily making it legible - this is what it means to build intuition
      • Examples can help fit things in to the bigger picture, they can motivate the ideas, and locate where they fit in to the student’s knowledge graph
      • By giving the student a pool of motivating examples, they can often generate the ideas themselves by generalising from the examples
      • Examples can bridge the gap from “understanding the knowledge in the abstract” to “understanding where to use these ideas, and where they should come up”
    • How to use them?
      • Often when giving a point, I’ll give a micro-example to give context to it - eg “sometimes straight lines aren’t enough to model data, eg with a quadratic”. This should be quick and effortless, the example should make immediate sense to the students, with 0 inferential steps
        • (I can’t believe it’s taken me 18 posts to get to my first nested example :( )
      • Examples can be good at the start, to motivate things and show the questions we’re trying to answer. It can be good to give an example, and then constantly refer back to it as we generalise the example into a concept
      • After introducing a complex idea, go through a long example and illustrate which parts of the example embody the complex idea
      • Use examples to illustrate the importance and relateability of an idea - eg if explaining how to think about good planning to students, give an example of a student with a deadline crisis that they missed - everyone relates to this
    • Examples contain a lot of information, so the idea of information prioritisation applies strongly here - tell the students what to pay attention to in the example, and why it’s interesting
  • Visual information and diagrams
    • Often tacit knowledge manifests as a literal picture in my head - draw this!
    • This is another good alternate communication channel
    • Our visual memory and processing is often much better than our abilities with language - this can work well for clarified confusing and complex parts
  • Pacing
    • An easy mistake that I often fall into is to give a section the amount of time it takes me to say it. I convert the ideas into words, and just read through what I’ve come up with.
      • This is the fallacy of viewing teaching as giving an information stream! Time should be allocated for people to process and compress information, and they need more time for hard parts and less time for easy parts
    • This is hard to get right intuitively - when you understand an idea, it feels easy!
    • A good trick: make a high-level summary, and rate each section out of 5 for difficulty. Then go and explicitly give more time to those sections - eg add more intuitions, say the key ideas more, give more examples
      • Note - pacing doesn’t mean the speed at which you speak, it’s about the time you give to different ideas!
      • Note that difficulty =/= importance. If one part is hard, but unimportant, cut it. Or give a brief overview, explain the important idea, and say “don’t sweat the details”
    • Another trick - explicitly tell students which ideas are important and worth paying attention to, and which aren’t
    • If doing a practice run (which you totally should), regularly check in with the test audience about pacing - the default state of the world is that you get pacing wrong
    • It's key to get pacing right - people zone out in slow, easy sections, and get lost in fast, hard sections. Your job as the teacher is to keep as many people as possible absorbing information at the optimum rate.
    • It’s very hard to give accurate time estimates for things - my trick is to have a bonus section at the end which I’ll cut if need be, and to pace in the moment according to my existing notes and my intuitions
  • Understand the mindset behind a question
    • When somebody asks a question, the default response is to answer it. This is a failure to be deliberate! The student asks a question because they’re confused about something, and your goal is to resolve that confusion - answering the question directly is just a means to an end.
      • This is an important distinction, because often questions are weird. They’re confused, or don’t quite make sense, or are asking about unimportant things. This manifests, for me, as the student’s mind not making sense. And it’s easy to get frustrated, or just to answer the dumb question directly. But this is ineffective.
        • A related effect - somebody asks a question that isn’t the real question they want to ask. Eg, a student at a university open day who asks “how many A-Levels did you do?”, when what they really care about is “how many A-Levels do I need to do to get in”
      • Your goal should be to understand the state of mind from which that question made sense - once you’ve done this, you can often resolve the confusion directly, or answer the question they really care about.
        • Do this by asking clarifying questions, trying to answer and saying “did that answer your question?”, giving them several interpretations of what they’re really asking and asking whether any resonate, paraphrasing the question back to them, etc.
        • The main trigger to look for here is a note of confusion - the question feeling a bit off, or out of nowhere, something isn’t quite making sense.
        • It’s a delicate balance between doing this and moving on with the talk - try to gauge whether many people share the same confusion, if not, just move on
    • Often doing this can uncover Pedagogical Content-Knowledge, common ways that people misunderstand the ideas you’re teaching. It’s super valuable to collect these, because then you can recognise them in future and dissolve them directly.
  • Illusion of transparency
    • A consequence of the typical mind fallacy - it’s easy to think you’ve clearly communicated knowledge when you really, really haven’t. As a consequence, you need to put a lot of effort into being grounded and calibrated - because often a confused audience won’t feel like a confused audience to you.
    • Ask questions! Especially ones that highlight the key ideas, eg “how to do easy thing X” or “what was the key idea in here?”
    • Do hand-polls - ask people to indicate their understanding by putting their hand up high if it’s clear, and low if it’s less clear. This is a good technique, because most people will actually do it, unlike “does anyone have any questions?” or “is this making sense to people?”
  • Seek feedback and iterate
    • Teaching is hard. You’re fundamentally trying to convey tacit knowledge, via lossy and low bandwidth communication channels, into an alien mind that you have very limited access to. The default state of the world is that you suck at this
    • The solution is to regularly ask for specific, actionable feedback and calibration, and to actually put meaningful effort into updating on this! Feedback is one of the main ways you can better understand the mind of a student.
Teaching 1-on-1

Practicing tutoring and explaining things one on one can often be more valuable! I think a great use of time for most students is to do tutoring - it’s pretty fun, you get paid decently, and you get way better at explaining ideas. And the ability to explain an idea clearly in a conversation is an amazingly applicable skill - I use this all the time in daily life.

The main difference is that it’s a lot easier to get them to be active, and it’s much easier to adapt the pace and difficulty well. Essentially, invert all of the ideas in my post on how to learn from conversation

  • The key technique is asking the student to paraphrase what you’ve said back to you
    • This forces them to be active, and to process information
    • It identifies errors, and helps you to correct them
      • Often you can then recognise th
    • It helps build a model of what’s going on in their mind
    • It helps you calibrate the difficulty and pacing
    • If you’re a tutor who doesn’t do get the student to this, I think you’re missing out on a major free win
      • This is also super effective when explaining ideas to friends, though can seem a bit rude
  • Get the student to tell you the key points/ideas in what you’ve said
  • Get the student to generate examples, especially typical examples
  • Here, understanding the mindset behind the question is even more important. You should always do this when they ask a question, especially if they seem dissatisfied with your answer.
Teaching Maths
  • It’s easy to neglect the tacit information - the intuitions, underlying concepts, motivations. This is terrible. One of the most important parts of teaching maths well is to convey this high-level overview
  • Every proof can be heavily compressed. Most proofs have some key ideas, followed by repeatedly doing the obvious thing. “Repeatedly doing the obvious thing” will inevitably be compressed in the student’s mind, so you should skip saying it at all, and just give the key ideas
  • Examples, especially motivating examples, are incredibly important. It’s really, really hard to learn a new concept without having a clear motivating example in mind.
    • Examples teach tacit knowledge well - they illustrate what you can and can’t do, and why you care about ideas
  • After learning rigour for a while, you’ll end up with post-formal intuitions, where you mostly ignore rigour and think intuitively, but can drop into rigour if need be. Most of the cognitive labour in maths is reaching this point, and a good teacher will try to give as much of the post-formal intuition as possible
  • Maths, especially pure maths, is often formalising an intuition. Probability is the formal study of uncertainty. Groups are the formal study of symmetries. Topology is the formal study of continuous deformations (things which don’t rip or glue). Pointing this out is vital
    • A good way to find these is to notice which questions the topic newly lets you answer. This is a great way to motivate things!
  • Diagrams are awesome
  • Often you begin being able to answer a type of question with a lot of tacit knowledge, and are expected to pick all this up with examples. Often 80% of this can be captured in an explicit algorithm - this is a great way to make tacit knowledge legible.
Teaching Applied Rationality/Life Advice
  • These are much more about tacit knowledge than explicit, so these should be done in a workshop format with a big focus on exercises
    • Emphasise that everything is highly personal and subjective - all ideas should be adapted to your mind and your circumstances
    • Pairing people up works well for getting them to actually do the exercises!
  • Ideally, boil down the tacit knowledge to a rough algorithm, and alternate explaining steps and getting the students to practice them
  • The impact of the class is mostly students retaining key insights and mental habits - what the “time when I should apply this idea” feels like from the inside. This is the small fraction of information they’ll retain. Thus it’s your job to boil down the idea to these habits, say this explicitly, and structure the class to reinforce them
  • Much of the impact comes from the students taking action after the class - this is hard! You want to emphasise this, and minimise barriers
    • Give time for the students to generates lists of ways to apply the ideas, and how they’re relevant in their day to day life
    • Give time for them to set reminders for actions taken after the class
  • Make it feel actionable - it’s easy to think an idea is important, but for it to feel abstract. Eg, to think that prioritisation is a good idea, but to never get round to it.
    • Emphasise how the idea fits into everyday life and everyday problems
    • Give a lot of examples of how to use it - this can form connections like “oh! I never thought of using it for that”
      • This conveys the tacit knowledge of when to use the idea!
  • Emphasise relatability and importance - give examples of a bad situation where the technique wasn’t applied, and make it feel visceral and relatable
Conclusion

If you’re planning on teaching something in the future, I hope these thoughts were useful! But even if not, I think these skills transfer excellently to explaining things in everyday life. And that thinking about teaching can make you a much more effective learner.

I find that often, as a student, I can help the teacher be more effective by asking the right questions - asking them which information is the most important, checking that my understanding is correct by paraphrasing back, asking them for the motivations and higher-level picture. The feeling of “something not fitting into my knowledge graph well” can be made into a pretty visceral one. And realising the habits of students that hinder them from learning, like being passive instead of active, and not trying to do information compression themselves, can help me recognise when I fail to do those things!



Discuss

Дискуссии

События в Кочерге - 28 августа, 2020 - 19:00
Чем обычная кухонная беседа отличается от вдумчивого и эффективного диалога? Ежемесячный формат с дискуссиями и повышением культуры общения.

How hard would it be to change GDP-3 in a way that allows audio?

Новости LessWrong.com - 28 августа, 2020 - 17:42
Published on August 28, 2020 2:42 PM GMT

GDP-3 currently only works on text. If OpenAI would desire to make it work with similar performance for audio, how much work would that likely be?



Discuss

Zibbaldone With It All

Новости LessWrong.com - 28 августа, 2020 - 14:37
Published on August 28, 2020 11:37 AM GMT

Less Zettelkasten! More Zibbaldone!

A Zettelkasten requires you to intricately interconnect and crosslink your thoughts, figuring out exactly how each concept relates to every other concept. A Zibbaldone is writing down whatever random thing comes to mind - an omnisubject diary of sorts. That's what people are throwing into their Zettelkasten, with their careful annotations and interrelations. What if everyone could access everyone else's Zettelkasten, and interrlate them into a glorious omniscient noospheric substrate? Wow! Whatta thought!

Buuuuuuullshit. This is semantic web stuff all over again - and right at the cusp of it having been clearly and entirely outmoded by automatic natural language processing. We don't need to manually mark up pages when Google will index them all for us anyway! Humans might think in that chaotic, random Zettelkastian way, but explicit sentence composition and task completion is temporally linear, and so we need to output our thoughts in much the same way. Not that we wouldn't love to manually annotate our every passing thought, but god, who has time for that?

What we're really missing is a tool that will take whatever random trash I throw out of my brain, figure out how it relates to historical junk I threw out of my brain, and point out the connections for me. Hell, ideally mid-composition: a modern ersatz Clippy pops up - "Hey, it looks like you're talking about Wittgenstein again! Do you want me to autofill what you already think about it, or have you changed your mind about the topic?"

Not only that: once you get your thoughts out of your head bereft of order, an AI could rephrase your thoughts back to you more clearly, more concisely, help interrogate what you mean, point out logical contradictions, challenge your ideas. We need only Zibbaldone out our stream of consciousness and it can be autonomously interrlated into a glorious omniscient noospheric substrate! Wow! Whatta thought!

...just ignore the risks of mentally coupling with an opaque box, whose biases you don't understand, whose intentions may be short term and not long term, who may network you with exactly the wrong people. Zibbaldone to Zettelkasten mapping seems like a middling-difficulty problem with existing tools, and whoever solves it is going to make a lot of money. When we can auto-generate a Zettelkasten from a blog - or dozens of them - you'll have a whole world of brilliance to tap into!

...or bias. Or noise. Or exactly the sort of thing you're trying to escape from - the internet, where the awful people are, with the Bad Opinions. And what you end up is just another wiki to mindlessly trawl through, to process, to "integrate". I'm starting to think the important part of a Zettelkasten isn't making the links for your future self so much as training yourself to notice the links at all. The Zettelkasten's just external practice for what should be an internal and subconscious process - seeing the fnords, noticing the interconnectedness, it's just training to be a holistic discordian - getting you past the madness of your surface layer thoughts and into the deeper crystalline method.

Of course, everyone will think you're a little crazy - and of course, they'll be a little right. But when the going gets weird, the weird turn pro, and when insanity's the zeitgeist, nothing's more professional than crazy talk. Have fun making bank on the Zibbaldone-Zettelkasten mapping, crafting neurological parasites, offloading the part of our brain with connections and opinions to an external cognitive artifact! No way that can go wrong.

Really, the harder problem's going the other way - give me a web, which path should I walk? What is the narrative of a human mind? You can't read a Zettelkasten effectively, it has too many strands and streams and errant thoughts; you have to craft a narrative out of it, even if you're consuming it out of order, but figuring out how to knit the threads together into a cohesive chunk for consumption's the hard part. What's the mapping then? Zibbaldone to Zettelkasten to... Roman-Fleuve, ministructure to the metastructure? Zettel-fleuve, Zibbal-fleuve? Zittal, Zabbal, Zettaldone, Zibbalkasten, - god, we're all turning into lunatics aren't we? The jargon's making zibbering lunatics of us all - just write something down, anything, throw it to the aether and be zibbaldone with it all.



Discuss

C̶a̶m̶b̶r̶i̶d̶g̶e̶ Virtual LW/SSC Meetup

Новости LessWrong.com - 28 августа, 2020 - 05:45
Published on August 28, 2020 2:45 AM GMT

The September Cambridge Less Wrong / Slate Star Codex (RIP) meetup will be held online due to the plague.

Hangouts link



Discuss

RE "On Bullshit" and "On Truth," by Harry Frankfurt

Новости LessWrong.com - 28 августа, 2020 - 03:44
Published on August 28, 2020 12:44 AM GMT

Salticidae Philosophiae is a series of abstracts, commentaries, and reviews on philosophical articles and books.

Harry Frankfurt asks, “What is bullshit, anyway?” Also, “What is truth?” but we all know that book proposal wouldn’t have flown except as a companion to the first one.

Highlights
  • Something can be true, and still be bullshit.
  • Something can be a lie, and yet not be bullshit.
  • Bullshit is that which is (1) unconcerned with truth and (2) intended to change attitudes rather than beliefs.
  • Truth is useful to us as individuals and as societies
  • Truth-seeking and truth-telling must be rewarded and their inverse must be punished.
  • Truth is truth whether or not anyone believes it or even knows it.
New or uncommon terminology
  • On Bullshit is described as a prolegomenon to On Truth, or an extended introduction that serves to discuss and interpret the work in a manner that is more exhaustive than the typical introduction.
Book-by-bookOn Bullshit

There is not much literature on bullshit, and no "theory of bullshit" or rigorous analysis thereof. This is in large part because we all assume that we recognize and evade bullshit pretty well.

According to Max Black, humbug is essentially a (false) statement made, not to convince you about that thing, but to convince you of something else. For example, one might make blatantly and obviously exaggerated or otherwise false statements about U.S. history not to convince another of these things, but to convince another of one's patriotic fervor.

Starting from this definition of humbug, Frankfurt makes a number of comparisons and caveats that might be useful:

  • Bullshit may be made carelessly, and we could easily compare bullshit to shoddy goods.
  • Shit is excreted, not crafted. However, advertising can be carefully-crafted bullshit.
  • Similes are not lies, but they can be made too thoughtlessly. In their own way, they can be bullshit.

Frankfurt argues that bullshit is, to start with, deliberate misrepresentation. Some say that lying requires intent; others, that any false statement is a lie. Bullshitting, however, is not exactly the same as lying. Indeed, bullshit can be true. Frankfurt's position is that bullshit is distinguished not by its truth or falsity, but by a disregard for the truth; as he puts it, honest folk and liars are playing the same game, to convey the facts or to obscure them, but the bullshitter is playing another game entirely.

Truth-tellers and liars are both concerned with changing your beliefs; a bullshitter is concerned with changing your attitude.

  • "Someone who ceases to believe in the possibility of identifying certain statements as true and others as false can have only two alternatives. The first is to desist both from efforts to tell the truth and from efforts to deceive. [...] The second alternative is to continue making assertions that purport to describe the way things are, but that cannot be anything but bullshit." [pg 61 para 2]
  • "Just as hot air is speech that has been emptied of all informative content, so excrement is matter from which everything nutritive has been removed. Excrement may be regarded as the corpse of nourishment, what remains when the vital elements in food have been exhausted. In this respect, excrement is a representation of death that we ourselves produce and that, indeed, we cannot help producing in the very process of maintaining our lives." [pg 43 para 1]
On TruthIntroduction

This is a sequel to On Bullshit, which addresses an oversight of his: the author failed to make any argument as to why the truth is important, and bullshit is therefore reprehensible. This book is about why truth is important.

There is lots of bullshit but it hasn't destroyed civilization, so some people think that truth isn't important. Some people even refuse to admit that there is such a thing as truth. though these people are very silly (not least because they tend to represent themselves as truly holding this belief). The book therefore assumes that there is an "objectively meaningful or worthwhile distinction to be made between what is true and what is false," and concerns itself solely with addressing whether this distinction matters outside of academia.

He spends more that a tenth of the book explaining what he's doing and why he's doing it.

Chapter I

Truth is useful to us. Societies cannot function without fostering truth. Both individuals and groups must know facts and as societies become more complex they must know more facts, and more accurately (while many individuals, it must be said, can remain free riders).

Postmodernists reject the idea that truth has objective reality or value, at least as perceived by us; our view of the truth is determined by constraints that have been imposed upon us by personal and social environments and histories. It is interesting that postmodernism does not exist (in this form) in medicine, physics, and other fields whose assertions are easily testable. Even in history, there must be objective facts: "They will not say that Belgium invaded Germany," the author reports Georges Clemenceau as saying.

Chapter II

Even if some value statements are not verifiable, they can generally be connected back to facts that can be discussed. Knowing the facts of the matter lets us determine whether we ought to value the things that we do, or whether other goals and activities might better accomplish our terminal values.

  • Healthy societies must reward truth-finders and punish truth-obscurers.
  • Having facts is not enough to succeed (you must use them properly), but not having facts prevents you from taking any action at all.
Chapter III

One might argue that we could just not care about this need for truth. Spinoza argued that we cannot help but care, because of love, which is "nothing but Joy with the accompanying idea of an external cause." essentially an experience that broadens one's understanding of oneself and improves one's capacity for perfection. or (in the author's words) "the way that we respond to something that we recognize as giving us joy." Additionally, joy is the experience of being ennobled or otherwise improved (and, preferably, knowing this). Therefore, truth gives us joy, because it improves us, and because we wish to preserve and keep nearby that which we love, we will seek to preserve truth.

Chapter IV

When we act, we interact with reality, and we have a desire or at least an expectation regarding the outcome of our action. To the degree that we lack truth, we are disconnected from reality and that desire or expectation may be thwarted.

  • It is always better to face uncomfortable truths than to hide away from them, because if we do not confront them then, one day, we will be confronted by them.
  • Without truth, we are blind. We might not run into trouble immediately, but we will do so inevitably.
  • "The relevant facts are what they are regardless of what we may happen to believe about them, and regardless of what we may wish them to be. This is, indeed, the essence and the defining character of factuality, of being real: the properties of reality, and accordingly the truths about its properties, are what they are, independent of any direct or immediate control by our will." [pg 55 para 2]
Chapter V

Truth fosters trust. Honesty is the foundation of society, while dishonesty undermines social fabric. Even the capacity for self-recognition (or self-awareness, we might say) ultimately depends on our relationship with the truth. If we do not know the world, then we cannot know ourselves.

  • If someone starts getting into etymology as part of some Deep Explanation, then prepare for a torrent of bullshit.
  • Immanuel Kant, "On a Supposed Right to Lie from Altruistic Motives": "A lie always harms another; if not some particular man, still it harms mankind generally.
  • Michel Montaigne, "Of Liars": "If we did but recognize the horror and gravity of [...lying], we would punish it with flames more justly than other crimes."

Even the capacity for self-recognition (or self-awareness, we might say) ultimately depends on our relationship with truth. If we do not know the world, then we cannot know ourselves.

Comments

On Bullshit argues that bullshitting doesn't necessarily undermine society, at least not up to a point, but I I would expect a hypothetical society with even slightly less bullshit than ours to function more smoothly. I also disagree with the position that truth intrinsically gives us joy. Many of us love bullshit more than truth.

Frankfurt says that lying is bad at its core because the liar "tries to impose his will on us," even if it is for our own good, but he fails to argue that this in itself is bad. More convincing is Frankfurt's argument that we are being pushed into another world insofar as our beliefs are false, but what if the lie is believed on a large scale? Then we would be isolated by believing the truth. He also argues that the liar is personally isolated, and cannot even speak of that isolation, but this is untrue if the liar has partners.

Favorite passageAs conscious beings, we exist only in response to other things, and we cannot know ourselves at all without knowing them. Moreover, there is nothing in theory, and certainly nothing in experience, to support the extraordinary judgment that it is the truth about himself that is the easiest for a person to know. Facts about ourselves are not peculiarly solid and resistant to skeptical dissolution. Our natures are, indeed, elusively insubstantial--notoriously less stable and less inherent than the natures of other things. And insofar as this is the case, sincerity itself is bullshit. [On Bullshit, pg 66 para 2]Author biography

Harry G. Frankfurt is Professor of Philosophy Emeritus at Princeton University. His books include The Reasons of Love (Princeton), Necessity, Volition, and Love, and The Importance of What We Care About.

Philosophers & works mentioned

Philosophers given significant attention include:

  • Max Black, a British-American philosopher who, for some reason, has a longer article on the Unitarian-run New World Encyclopedia than he does on Wikipedia, even though the former takes its articles from the latter before it edits and builds upon them.
  • Immanuel Kant, a German philosopher who argued that reason is the basis of morality, and drew attention to the difference between the world-as-it-is and the world-as-it-appears-to-us.
  • Michel Montaigne, a French philosopher of the Renaissance period who wrote on child education, psychology, and other topics, and popularized (but did not invent) the essay format.
  • Baruch Spinoza, a Jewish Portuguese philosopher who is best known for his writings on God, which have gotten him labeled as everything from a pantheist to an atheist.
Other articles & books on this subject

Discuss

Notes on "The Anthropology of Childhood"

Новости LessWrong.com - 27 августа, 2020 - 20:11
Published on August 27, 2020 5:11 PM GMT

Crossposted from The Whole Sky.

I read David Lancy’s “The Anthropology of Childhood: Cherubs, Chattel, and Changelings” and highlighted some passages. A lot of passages, it turns out.

[content note: discussion of abortion and infanticide, including infanticide of children with disabilities, in “Life and Death” section but not elsewhere]

I was a sociology major and understood anthropology to be basically “like sociology, but in Papua New Guinea.” This is the first cultural anthropology book I’ve read, and that was pretty much right. I found it very accessible as a first dive into anthropology. The first chapter summarizes all his points without the examples, so you could try that if you want to get the gist without reading the whole book.

I enjoyed it and would recommend it to people interested in this topic. A few things that shifted for me:

  • I feel less obliged to entertain my children and intervene in their conflicts. We don’t live with a tribe of extended family, but my two children play with each other all day, which is how most people throughout time have spent their childhoods. Lancy isn’t a child development expert, but I buy his argument that handling conflict (for example about the rules of a game) is a skill children need to learn, rather than having conflicts always mediated by adults.
  • Even though it doesn’t change anything concrete, I feel some relief that not having endless patience for toddlers seems to be normal. Except where families were very isolated, it’s not normal in traditional societies for one or two adults to watch their own children all day every day. And childcare has traditionally looked mostly like “being sure they don’t hurt themselves too badly.”
  • It surprised me that childcare by non-parents was so common. Some more modern views treat women’s childcare work as basically free, traditional cultures have valued women’s labor enough that the society wants to free up their time from childcare. It was striking to me that the expectation that stay-at-home mothers will be responsible for all childcare was a relatively short historical blip. But of course, having childcare done by teenagers and grandmothers requires that those people’s time be available, which usually isn’t the reality we live in.
  • I was surprised at how apparently universal it is for fathers to be uninvolved.

I’m a little unclear on how valid Lancy’s conclusions are or how much data they’re based on. It seems like an anthropologist could squint at a society and see all kinds of things that someone with a different ideology wouldn’t see.

Big caveat that what Lancy is describing is traditional, non-industrialized societies where children are expected to learn how to fit into the appropriate role in their village, not to develop as an individual or do anything different from what their parents and ancestors did. He stresses that traditional childrearing practices are very poor preparation for school. Given that I want my children to learn things I don’t know, to think analytically, etc, the way I approach learning is very different from how traditional societies approach it.

One complaint is that Lancy periodically complains about how much money Western families spend on fertility treatments, medical care for premature infants, etc. He argues that the same money could be used to provide adequate nutrition for many more children in the societies he’s studied. I’m sympathetic, but assuming that families would donate this money if they weren’t spending it to have a baby is not realistic. I see cutting luxury spending as a much more feasible way that people might do some redistribution.

And now, my notes:

Views of childhood

As in many areas of research, the children who have been studied by academics are mostly from WEIRD ("Western, educated, industrialized, rich, democratic”) populations. Thus our understanding of good or normal childrearing practices is very different from how children have typically been raised. Lancy contrasts modern childrearing norms with those of traditional agrarian or forager societies.

Lancy contrasts neontocracy (where babies and children are most valued) with gerontocracy (where elders or ancestors are most valued). I can think of ways our society isn’t very good for children, but I agree that compared with traditional societies, we spend a lot of attention and money on children. (Albeit sometimes by micromanaging them, while Lancy would rather have them figure out more for themselves as children have historically done.)

Even studying children is a strange thing to do in most societies. “Examples of children treated as lacking any sense, as being essentially uneducable, are legion in the ethnographic record.” “Anthropologists interested in children are treated in a bemused fashion; after all, why bother to observe or talk to individuals who ‘don’t know anything’?” (Lancy 1996: 118; also Barley 1983/ 2000: 61)“

“Infants were widely seen as insensible. Almost like plants, their care could be rudimentary”

Traditional societies have two broad patterns toward young children: “One response is ‘benign neglect’– everyone waits until the child can talk sensibly before acknowledging its existence. A second typical response is to aggressively humanize the child, including ruthless suppression of all ‘sub-human’ tendencies (e.g. bawling, crawling, thumb-sucking).”

Europeans were of the second view:

“Like wild men [or beasts], babies lacked the power to reason, speak, or stand and walk erect. [They were] nasty, brutish, and dirty, communicating in wordless cries, grunts, and screams, and were given to crawling on all fours before they could be made to walk like men … Left to their own devices, they would remain selfish, animalistic, and savage. Parents believed they had to coerce their babies into growing up, and they expected protests and resistance. (Calvert 1992: 26, 34)”

“The Puritans were perhaps the first anxious parents, fearing they might fail and their children would turn out badly.”

“We now take for granted the “need” to stimulate the infant through physical contact, motherese, and playing games like peek-a-boo to accelerate physical and intellectual development. Contrast these assumptions with the pre-modern objective of keeping babies quiescent so they’d make fewer demands on caretakers and not injure themselves (LeVine et al. 1994).”

“Much of what we think of as the routine duties (e.g. reading bedtime stories; cf. Lancy 1994) or expenses (e.g. orthodontics) of modern parents are completely unknown outside modern, mainstream societies.”

“150 years ago, the idea of the useful child began to give way to our modern notion of the useless but also priceless child (Zelizer 1985). Children become innocent and fragile cherubs, needing protection from adult society, including the world of work. Their value to us is measured no longer in terms of an economic payoff or even genetic fitness but in terms of complementing our own values – as book lovers, ardent travelers, athletes, or devotees of a particular sect.”

“Known as the “largest children’s migration in history,” so-called “orphan trains” carried about 200,000 children (Warren 2001: 4) from orphanages and foundling homes in eastern coastal cities to families in the Midwest (Kay 2003: iii) and West. The orphan trains continued until 1929 (Warren 2001: 20), which indicates how very recently our fundamental conception of children as chattel changed to viewing them as cherubs.”

Anne of Green Gables is a story about this dynamic in Canada — the family was expecting to adopt a boy who could serve as an unpaid farmhand, but got a girl orphan by mistake.

Who cares for children?

Surprisingly to me, in traditional societies it’s usually not mothers.

In the early days of infancy, of course, breastfeeding necessitates keeping mother and baby convenient to each other. “Nearly all societies hold very strict views on the necessity for almost constant contact between a mother or other nurturing adult and the infant. Infants are fed on demand, carried constantly, and sleep with their mother. Young mothers are severely chastised for any lapse in infant care. However, once the infant begins to walk, it immediately joins a social network in which its mother plays a sharply diminished role – especially if she’s pregnant – and its father may play no role at all.”

On the saying “it takes a village to raise a child”: “If one actually looks at real kids in real villages, either one sees infants and young children in a group of their peers, untended by an adult, or one sees a mother, or a father, or an older sister, or a grandmother tending the child. These helpful family members are referred to in anthropology as ‘alloparents.’ The rule governing their behavior would not necessarily be ‘Everyone’s eager to have a hand in caring for the child,’ but, rather, ‘Whoever can most easily be spared from more important tasks will take care of the child.’ And the next rule we might derive from our observations might be, “The mother is often too busy to tend to the child.” At the same time, babies are not simply passive recipients of care. They not only look cute, they beguile caretakers with their gaze, their smiling and their mimicry (Spelke and Kinzler 2007: 92). While alloparents may want to minimize their effort (Trivers 1974) in caring for the child, the very young have an arsenal of tactics they can deploy to secure additional resources (Povinelli et al. 2005).”

“Weisner and Gallimore examined hundreds of ethnographies in the Human Relations Area Files (HRAF) archive and found that, in accounts of childcare, 40 percent of infants and 80 percent of toddlers are cared for primarily by someone other than their mother, most commonly older sisters (Weisner and Gallimore 1977).”

“Three-year-old children are able to join in a play group, and it is in such play groups that children are truly raised” (Eibl-Eibesfeldt 1989: 600).”

“Once the infant has been judged worthy of rearing, it will be displayed to a community eager to interact with it. In particular, its older sisters will be in the forefront of those wanting to share in the nurturing process. The circle of caretakers may gradually widen to include aunts, grandmothers, and, occasionally, the father. Even more distant kin can be expected to cast a watchful eye on the child when it is playing on the ‘mother-ground’ (Lancy 1996: 84). Indeed, the toddler must seek comfort from relatives as it may be abruptly weaned and forcibly rejected by its mother as she readies herself for the next child.”

In a large polygynous household the author visited in Liberia, even after a few weeks he was unable to figure out which children belonged to which mothers: “I was stymied because the children, once they were no longer attached marsupial-like to their mother’s body with a length of cloth, spent far more time in each other’s company and in the company of other kin, particularly grandmothers and aunts in nearby houses, than with their mothers. And as far as the chief was concerned, I just had to assume that since these were his wives, the majority of the children in the vicinity must be his as well. Aside from dandling the occasional infant on his knee during the family’s evening meal, I never saw him enjoy more than the most fleeting interaction with a child.” Later, “I began to see their family arrangements and childcare customs as neither unusual nor exotic, rather as close to the norm for human societies, and, simultaneously, to see the customs of the middle-class Utah community I live in now as extraordinary.”

Older sisters are often alloparents:

“Across the primate order, juvenile females show great interest in infants (Hrdy 1999: 157), and it is not hard to sustain an argument that their supervised interaction with younger siblings prepares them for the role of motherhood (Fairbanks 1990; Riesman 1992: 111). The weanling’s need for mothering corresponds to the allomother’s need to mother.”

This seems to be true in other primates as well (though I do imagine researcher bias could interpret some kinds of carrying around a stick as ‘doll play’ depending on the gender of the young chimp.)

Several studies have documented the gender bias in “baby lust” (Hrdy 1999: 157). Females show far more interest in babies, images of babies, and even silhouettes of babies than do males. In fact, there’s some evidence that young chimp females will cradle, groom, and carry around a “doll” (a stick or a dead animal) in the absence of a live infant (Kahlenberg and Wrangham 2010: 1067).

“In Uganda in 2003, I observed and filmed numerous primate species and, after resting, eating, and play, “baby-trading” is the most common occupation. Often I observed what amounted to a “tug-of-war” between the nursing mother and her older daughters for possession of the infant, which may lead to what Sarah Hrdy (1976) referred to as “aunting to death.” By contrast, mothers tend to discourage interest shown by juvenile males in their offspring (Strier 2003).12

“Aunting to death” sounds familiar to me. When Lily was born, we lived with Jeff’s family including his two sisters. They would literally race each other to the baby each morning when I came downstairs with Lily, as each aunt tried to arrive first for baby cuddles.

Boys are not seen as good caregivers:

“Dozens of studies have documented the heightened likelihood of sensation-seeking (Zuckerman 1984) or risk-taking by adolescent primate males in groups. Demographers have identified an “accident hump” in mortality curves for male primates, including humans, during puberty (Goldstein 2011).”

"I had a personal epiphany regarding the inadvisability of assigning boys as sibling caretakers in May 2007 as I stood on a busy street in front of the Registan in Samarkand. Two boys were pushing baby carriages in the street, just barely out of traffic. The street sloped downward and the lead carriage-pusher began a game of chicken, releasing his grip on the bar, then rushing after to grab it as the carriage rolled away on its own. This game was repeated with longer intervals between the release and retrieval."

Children need less oversight in less dangerous environments:

“Tether-length is definitely a useful concept in observing human mother– toddler interaction (Broch 1990: 71–72). As Sorenson discovered in a Fore village, the infant’s “early pattern of exploratory activity included frequent returns to the mother. She served as the home base, the bastion of security but not as director or overseer of activities” (Sorenson 1976: 167). For the forest-dwelling Chewong, the tether is shorter. Toddlers are discouraged from wandering away from proximity to adults with “loud exclamations …‘it is hot,’ or ‘it is sharp,’ or ‘there are … tigers, snakes, millipedes’” (Howell 1988: 163).”

Swaddling makes children easier to watch:

“A swaddled baby, like a little turtle in its shell, could be looked after by another, only slightly older child without too much fear of injury, since the practice of swaddling made … child care virtually idiot proof. (Calvert 1992: 23–24)”

There is a chain of oversight:

“toddlers are managed by slightly older siblings, who are, in turn, guided by adolescents, while adults serve as rather distant “foremen” for the activity, concentrating, primarily, on their own more productive or profitable activity.”

The stereotype of grandmothers “spoiling” children is not unique to the West:

“[I]n the Mende view, grannies are notoriously lax with children. They are said to feed children upon demand and do not beat them or withhold meals from them for bad behavior or for failing to work … Children raised like this are said to grow up lazy and dishonest …”

In Rome, nurses were responsible for childcare in wealthy families:

“It was the nutrix [nurse] who … took responsibility for … early infant care: breast-feeding, powdering and swaddling, bathing and massaging, rocking and singing the child to sleep, weaning the child from milk to solid food … The nutrix, in fact, was only one of a sequence of child-minding functionaries who influenced the early lives of children.”

“Public attitudes in Europe reflect a view of the family that echoes the utopian ideals of the Israeli kibbutz from the mid-twentieth century. While the mother might be the primary caretaker during infancy, shortly afterward the child should be placed in a nursery with trained staff as she returns to her job. This policy is seen as beneficial to the mother’s self-esteem, the economy, and the child itself (Corsaro 1996; Dahlberg 1992; Eibl-Eibesfeldt 1983: 181). Publicly supported pre-school or daycare in the US has been blocked by the politically powerful religious right, which insists on keeping wives tied full-time to the kitchen and nursery.”

When childcare is a collective task, discipline is also collectivized:

“The mother must, however, accept the consequence that virtually anyone older than her child can scold or even discipline them (Whiting 1941). In societies like our own, where childcare is handled within the nuclear family and/or by professionals, the necessity for learning manners and kinship arcana is reduced. At the same time, we are often reluctant to concede to outsiders, even “professionals,” the right to discipline our young.”

Why do mothers outsource childcare?

“In a majority of the world’s diverse societies, women continue as workers throughout pregnancy and resume working shortly after the child is born. This work is physically demanding, so, for many, there is a peak period in their lives when they have the stamina and fat reserves to do their work and have babies. How many babies they successfully rear will depend heavily on their access to a supportive community of relatives who can help with household work, assist with childcare, and provide supplementary resources.”

How children are taught to relate to others

Contrasted with the emphasis on the mother-child bond in WEIRD society generally and especially in “attachment parenting”, traditional cultures may emphasize finding other caregivers:

“The baby’s cherub-like features aid the mother in her quest for helpers. Young mammals, generally, but especially humans, display a suite of physical features that seem to be universally attractive to others, and these features are retained longer in humans than in other mammalian species (Lancaster and Lancaster 1983: 35; Sternglanz et al. 1977). Also critical is the fact that human infants vocalize, make eye contact, and smile from very early on (Chevalier-Skolnikoff 1977) – unlike chimps, for example, whose mothers make more limited use of helpers. Mothers may not always rely on the inherent cuteness of their babies; they may take pains to showcase the baby – at least among close kin. The Kpelle mothers I observed didn’t stop at frequently washing and cleaning their babies. They oiled the babies’ bodies until they gleamed – an ablution carried out in public view with an appreciative audience. The Kaluli mothers studied by Bambi Schieffelin in Papua New Guinea not only hold their infants facing toward others in the social group – a practice often noted in the ethnographic record – but treat the baby as a ventriloquist’s dummy in having him or her speak to those assembled (Schieffelin 1990: 71). The Beng advise young mothers: Make sure the baby looks beautiful! … put herbal makeup on her face as attractively as possible … we Beng have lots of designs for babies’ faces … That way, the baby will be so irresistibly beautiful that someone will feel compelled to carry her around for a while that day. If you’re lucky, maybe that person will even offer to be your leng kuli. (Gottleib 1995: 24) When [Guara] neighbors visit … relatives – identified by kinship terms – are repeatedly indicated to the child. (Ruddle and Chesterfield 1977: 29) [Marquesan mothers] … spent much time calling the baby’s name, directing him to look and wave at others … directing three- to six-year-old siblings to play with him. (Martini and Kirkpatrick 1981: 199)”

“Samoan …toddlers were fed facing others and prompted to notice and call out to people. (Ochs and Izquierdo 2009: 397) From the moment a [Warlpiri] child is born … she will hear every day … for the next few years; “Look, your granny,”‘That’s your big sister, your cousin, your auntie.” In fact, they make up the bulk of verbal communication with babies and little children. (Musharbash 2011: 72)”

“There were numerous constraints put on young [Orissa India] mothers to prevent them from focusing too much attention on a new infant. Close, intimate mother-child bonds were viewed as potentially disruptive to the collective well-being of the extended family … In such families, much early child-care was organized so as to subtly push the infant away from an exclusive dependence on its mother toward membership in the larger group. (Seymour 2001: 15)”

How do parents learn to parent?

Partly through alloparenting as described above. Among other primates:

“While the benefits to the mother are obvious, allomothering daughters also clearly benefit by learning how to care for infants (Fairbanks 1990). A study of captive chimpanzees showed that females prevented from interacting with their mothers and younger siblings were themselves utterly incompetent as mothers (Davenport and Rogers 1970).”

I was surprised at how hard it was to feed a newborn - in my case I got help from the midwife, a lactation consultant, and the pediatrician, but traditionally advice would come from family and neighbors:

“Field and colleagues, working with Haitian immigrant mothers in Miami, find these mothers often have difficulty feeding their offspring, who are therefore hospitalized for dehydration and malnutrition at a high rate (Field et al. 1992: 183). I think it’s possible these young women immigrants lost the opportunity to learn how to care for infants from older women.”

Among the Fulani of West Africa:

“All women caring for their first babies will have had years of experience taking care of babies … under the watchful and sometimes severe eyes of their mothers, aunts, cousins or older sisters. The other women … will immediately notice, comment on, and perhaps strongly criticize any departure from customary behavior on the part of mothers. (Riesman 1992: 111)”

(Anthropologists traveling with their own children also get a lot of advice from locals.)

Nutrition

I hadn’t really thought about how much of life in traditional societies revolved around the essential, never-ending task of getting calories. There is often not enough to go around, and social differences can be observed through which children’s growth is stunted.

“A study of the Mende found that senior wives did have higher fitness while junior wives had fewer surviving children than their counterparts in monogamous unions (Isaac and Feinberg 1982). Similarly, in Botswana, children of more senior wives enjoyed nutrition and school attendance advantages (Bock and Johnson 2002: 329).”

The author recalls seeing “a picture of a mother holding on her lap a boy and girl of about the same age, possibly twins. The girl was skeletal, obviously in an advanced state of malnutrition, the boy robust and healthy. He sat erect, eyes intent on the camera; she sprawled, like a rag doll, her eyes staring into space. That picture and what it represented has haunted me ever since.”

Babies of the preferred sex are likely to be nursed longer and have higher survival rates.

Many folk traditions recommend foods for children, or diets for sick children, that are undernourishing or likely to be contaminated:

“Meat is usually among the foods kept from children. This is probably harmful, as a protein shortage, in particular, is often found in recently weaned children. However, malnutrition is rarely identified by parents as the root of a child’s illness. Katherine Dettwyler pointedly titled her study of the Dogon Dancing Skeletons, describing, in graphic detail, the horrific sight of severely malnourished children. She finds that, while the mothers are aware of something amiss, they attribute the problem to locally constructed folk illnesses and seek medicine from the anthropologist to effect a cure. When she tells them to provide the child with more food, they are skeptical. Children can’t benefit from good food because they haven’t worked hard to get it, and they don’t appreciate its good taste or the feeling of satisfaction it gives. Anyway, “old people deserve the best food, because they’re going to die soon” (Dettwyler 1994: 94–95). Yoruba mothers feed children barely visible scraps compared to the portions they give themselves. Good food might spoil the child’s moral character (Zeitlin 1996: 418; also true for the Tlingit – cf. de Laguna 1965: 17). The prescription for a sick child among the Gurage tribe in southwest Ethiopia is often the sacrifice of a sheep: “The flesh of the sacrificial animal is eaten exclusively by the parents of the sick child and others who are present at the curing rite; no portion of the meat is consumed by the patient, whose illness may well stem from an inadequate diet” (Shack 1969: 296).”

“Aside from a demonstrable shortage of food (Hill and Hurtado 1996: 319), under-nutrition may be attributable to customs that support a shortening of the nursing period, such as the belief by some East African pastoralists that certain babies nurse “too much” and should, therefore, be weaned early (Sellen 1995). On Fiji, nursing beyond one year is condemned as keeping “the child in babyhood [, leading to] a weak, simpering person” (Turner 1987: 107). The Alorese use threats to discourage nursing: “If you continue nursing, the snakes will come … the toad will eat you” (Du Bois 1941: 114).” (The WHO currently recommends breastfeeding until age 2 or beyond.)

While medical science considers the first milk (colostrum) to be especially beneficial to the newborn because of the antibodies it contains, folk tradition often withholds it from newborns: “In a survey of fifty-seven societies, in only nine did nursing begin shortly after birth (Raphael 1966).”

Spacing children

Contrasted with agricultralists who go for large families, “foragers adopt a “survivorship” reproductive strategy. Around-the-clock nursing and a post-partum sex taboo combine to insure long intervals between births, leading to lower fertility. Low fertility is offset by the attention bestowed on the few offspring, enhancing their chances of survival (Fouts et al. 2001).” Breastfeeding suppresses women’s fertility.

“Another way in which nature contributes to increasing IBI [inter-birth interval] is through post-partum depression following a miscarriage, stillbirth, or infant death. Binser notes that depression elevates cortisol and leaves the mother lethargic and sleepy, which may just serve to put off the next pregnancy until she has had a chance to recoup her vigor (Binser 2004). Nature is aided by culture in promoting longer IBIs through injunctions that militate against long intervals between nursing bouts. Frequent, round- the-clock nursing maintains high prolactin levels. The post-partum taboo on intercourse between husbands and wives also plays a critical role in spacing births.”

In other cases the mother is physically separated from her husband: “The wife may be lodged in a birthing or 'lying-in' house (Lepowsky 1985: 64), or secluded in her own home, until, in the Trobriands, 'mothers lost their tans and their skin color matched that of their infants' (Montague 1985: 89).”

In traditional societies, early sexual activity was less likely to result in pregnancy because adolescents were often malnourished and their fertility lower than we’d expect.

Which children are preferred

I had assumed that boys were always preferred in traditional societies, but it depends. The gender preference, or lack therof, is influenced by parents’ expectations of help their children will provide them with.

“There is a world in which children almost always feel “wanted” and where “there is no cultural preference for babies of either sex” (Howell 1988: 159). Infants are suckled on demand by their mothers and by other women in her absence. They are indulged and cosseted by their fathers, grandparents, and siblings. Children wean themselves over a long period and are given nutritious foods (Robson and Kaplan 2003: 156). They are subject to little or no restraint or coercion. Infants and toddlers are carried on long journeys and comforted when distressed. If they die in infancy, they may be mourned (Henry 1941/1964: 66). They are rarely or never physically punished or even scolded (Hernandez 1941: 129–130). They are not expected to make a significant contribution to the household economy and are free to play until the mid to late teens (Howell 2010: 30). Their experience of adolescence is relatively stress free (Hewlett and Hewlett 2013: 88). This paradise exists among a globally dispersed group of isolated societies – all of which depend heavily on foraging for their subsistence. They are also characterized by relatively egalitarian and close social relations, including relative parity between men and women (Hewlett et al. 1998).”

“One thorough study compared Hungarian Gypsies (matriarchal) with mainstream Hungarian (patriarchal) society. Gender preferences were as expected and behaviors tracked preferences. Gypsy girls were extremely helpful to their mothers and tended to remain at home longer than their brothers, helping even after marriage. They were nursed longer than their brothers, while Hungarian boys were nursed longer than their sisters. “Gypsy mothers were more likely to abort after having had one or more daughters, while Hungarians are more likely to abort pregnancies when they have had sons” (Bereczkei and Dunbar 1997: 18).”

“Names such as “Boy Needed” (Oghul Gerek) or “Last Daughter” (Songi Qiz) are common for girls. (Irons 2000: 230)”

Family structure

“We now realize that mothers, fathers, and children have differing agendas. The nursing child wants to be the last child his mother will ever have so that he can enjoy her care and provisioning exclusively. The father will be opportunistic in seeking mating opportunities and display a similar fickleness toward the provisioning of his offspring. He will, in other words, spread his investment around to maximize the number of surviving offspring. The mother has the most difficult decisions of all. She must weigh her health and longevity and future breeding opportunities against the cost of her present offspring, including any on the way. She must also factor in any resources that might be available from her children’s fathers and her own kin network.”

(Of course I can think of many loving and capable fathers, not least my own partner. But I was surprised that they seem to have historically played so little role in childrens’ lives.)

Polygyny is a common traditional way of structuring families, “the great compromise” between these competing interests.

“Estimates range from 85 percent (Murdock 1967: 47) to 93 percent (Low 1989: 312) of all societies ever recorded (about 1,200) having practiced polygyny.”

“Women in a polygynous relationship gain access to a higher-ranking, reliable provider at the cost of emotional strain in sharing resources (including the husband’s affection) with others. In one study, children of senior wives were better nourished than children in monogamous unions, who were, in turn, better nourished than children of later wives (Isaac and Feinberg 1982: 632). A woman must weigh the trade-offs between marrying a young man in a monogamous union or marrying an older man and joining a well-established household as a junior wife. Studies show that, if they choose monogamy, they enjoy slightly higher fertility (Josephson 2002: 378) and their children may be somewhat better nourished (Sellen 1998a: 341). However, they are, perhaps, more likely to be abandoned or divorced by their husbands.”

Both polygyny and monogamy have their pros and cons:

“In my fieldwork in Gbarngasuakwelle, I lived (as a guest) in a large, polygynous household and the tensions were palpable. This was seen as harmful to children. The shaman (village blacksmith in this case) came often to divine the cause and, using appropriate rituals (inevitably involving the sacrifice of a chicken), would attempt to ameliorate it (Lancy 1996: 167).”

“In Uganda, monogamy has led to less stable marriages. A man, rather than bringing a second wife into the household, now abandons the first wife and her children to set up a second separate household with his new mate (Ainsworth 1967: 10–11). A typical case among the Nyansongo in Kenya describes a mother, whose childhood was spent in a large polygynous compound where multiple caretakers were always available, who must cope alone in a monogamous household. She leaves her three-year-old to mind her six-month- and two-year-old infants as she performs errands like bringing the cow in from pasture. Unfortunately, the three-year-old is simply not mature enough for this task and is, in fact, ‘rough and dangerously negligent’ (Whiting and Edwards 1988a: 173).”

“As societies become more mobile and men migrate seeking employment, the likelihood that the male will abandon (or neglect) his family in the village in order to establish a new family in the city is increasingly high (Bucher and d’Amorim 1993: 16; Timaeus and Graham 1989). And, perhaps most common of all, women whose fertility is on the decline are replaced by younger wives in peak breeding condition (Low 2000: 325)”

“The abandoned spouse and her children may face severe difficulties. One might think that an obviously fertile woman would be a ‘catch,’ but ‘Having a child towards whom a new husband will have to assume step-parental duties diminishes rather than enhances a woman's marriageability’ (Wilson and Daly 2002: 307). “

“In the case of a young, pregnant widow, ancient Roman law permitted both annulment and the exposure of the infant in order to enhance her chances of remarriage (French 1991: 21). Raffaele describes an unfortunate case in a Bayaka13 foraging band in Central Africa:

Mimba had been in a trial marriage … her partner’s father had refused to pay the bride price and she had just been forced to return to her own family. She is two months’ pregnant, and it is a disgrace for an unmarried Bayaka woman to give birth” (Raffaele 2003: 129). Fortunately for Mimba, the tribe’s pharmacopoeia includes sambolo, a very reliable and safe herbal abortifacient, which she will use. Mimba will return to the pool of eligible mates and, hopefully, will find a family willing to pay the bride-price so their son can join her in raising a family – something she could not accomplish by herself.’”

“Studies in the USA indicate that living with a stepfather and stepsiblings leads to elevated cortisol levels, immunosuppression, and general illness (Flinn and England 1995)31 as well as poorer educational outcomes (Lancaster and Kaplan 2000: 196). Daly and Wilson find that a child is a hundred times more likely to be killed by a stepparent than by a biological parent (1984: 499).

Some form of fostering, adoption, or “child circulation” is practiced in many societies:

“Most commonly the child is transferred ‘to fulfill another household’s need for labor’ (Fée – Martin 2012: 220) as a ‘helper’ (Inuit – Honigmann and Honigmann 1953: 46). The request may be for a girl in families with a shortage of female labor (Kosrae – Ritter 1981: 46; Bellona – Monberg 1970: 132). On Raroia boys are requested as they can work in copra processing (Danielsson 1952: 120). On the other hand, the impetus may begin with a family that has a surplus of children (Bodenhorn 1988: 14), or children too close in age, or discord within the family; or as the means to defray a debt. Stepchildren are often moved out of the natal home to make way for the new parent’s biological offspring.”

Life and death

The topic that most surprised me in the book was traditional attitudes toward abortion and infanticide. I thought of life before birth control as “the bad old days” when women, perhaps not even understanding how babies are conceived, might be sentenced to a lifetime of childbearing and rearing against their wishes. I had never thought about how traditional societies actually handled unwanted babies.

“Data from a range of societies past and present suggest that from one-fifth to one-half of children don’t survive to five years (Dentan 1978: 111; Dunn 1974: 385; Kramer and Greaves 2007: 720; Le Mort 2008: 25). The first-century CE philosopher Epictetus cautioned, “When you kiss your child, say to yourself, it may be dead in the morning” (Stearns 2010: 168).

"Extrapolating from these figures I’d guess that miscarriages and stillbirths were also common by comparison with modern, post-industrial society. And I’d expect that if half the children died, then the majority were seriously ill in childhood. Indeed, in many villages studied by anthropologists the level of clinical malnutrition is 100 percent, as is the level of chronic parasite infestation and diarrhea. There are, then, ample reasons for withholding investment in the infant and maintaining a degree of emotional distance.”

“Humans have always had to cope with the loss of infants, and societies have developed an elaborate array of “cover stories” to lessen grief and recrimination (Martin 2001: 162; Scrimshaw 1984: 443). As discussed in the previous chapter, the primary strategy is to treat the infant as not yet fully human. Most importantly, if the baby is secluded initially and treated as being in a liminal state, its loss may not be widely noted.”

Some societies believed repeated miscarriages or stillbirths were caused by demons, and treated them with various attempts at exorcism. “It should be understood that these folk theories and treatments not only serve to dampen the sense of grief or loss but, more importantly, they deflect blame from the living. The Nankani have constructed an elaborate myth of the “spirit child not meant for this world” to explain away the tragedy of mother or infant death in childbirth and/or chronic infant sickness and, eventually, death (Denham 2012: 180). The alternative to, in effect, blaming the deceased child or “evil forces” is to blame the parents or other family/community member.”

“While new mothers may be evaluating the actuarial odds, we know that many are also suffering from post-partum depression or, less severely, detachment from and indifference toward their offspring. An argument can be made that this failure to bond immediately with the infant is adaptive in that it permits the mother to keep her options open, and also shields her emotionally from the impact of the infant’s death – often, a likely outcome (de Vries 1987a; Eible-Eibesfeldt 1983: 184; Hagen 1999; Konner 2010: 130, 208; Laes 2011: 100).”

“In the Himalayan kingdom of Ladakh, high-altitude living imposes an extra cost on the expectant mother who does farm-work throughout her pregnancy. Her infant’s life chances, owing to inevitably low birth-weight and other complications, are sharply reduced (Wiley 2004: 6). The worth of a new child in Ladakh will always be calculated as a tiny fraction of that of his fully mature, productive mother. While the mother’s health is closely monitored and she is treated with great solicitude, her infant’s fate is of less concern. Its death will be “met with sadness, but also with a sense of resignation … they are buried, not cremated like adults” (Wiley 2004: 131–132).”

“It is not unusual for the [Ayoreo] newborn to remain unnamed for several weeks or months, particularly if the infant is sickly. The reason given is that should the child die, the loss will not be so deeply felt. (Bugos and McCarthy 1984: 508)”

“Being a “calculating” mother is not synonymous with wickedness; on the contrary, it is adaptive behavior. While the well-to-do mothers in the first section seem to “live for their children,” in the next section, we discover just how recently these attitudes have become incorporated in Western society. We will trace the fluctuating value of infants in history and see that what we now consider horrible crimes were, in earlier periods, the principal means of birth control.”

In ancient Greece, “Illegitimacy was usually a death sentence. “Identity was given by the family, and without a recognized father and family, the child had no proper guardian (kurios) since its mother could not legally fulfill such a function. Without a father, the child had no true place in the patrilineal kin structure, no right to the family name” (Patterson 1985: 115). Until at least the end of the eighteenth century, any Venetian infant of questionable parentage would have been abandoned or destroyed (Ferraro 2008).”

“While the termination of the fetus or of the infant’s life is most often the parents’ decision and we’ve seen numerous possible reasons for this behavior, societies often legitimize that decision. Overpopulation, the burden on the community of a hard-to-raise child, the social disharmony created by illegitimacy: all give the society a stake in this critical decision. Ultimately, also, the community must value the life and emotional wellbeing of its experienced, productive adult females over any potential value a tiny infant might have.”

In foraging societies, “Both men and women face significant health and safety hazards throughout their relatively short lives, and they place their own welfare over that of their offspring. A survey of several foraging societies shows a close association between the willingness to commit infanticide and the daunting challenge “to carry more than a single young child on the nomadic round” (Riches 1974: 356).”

“The Inuit, among others, were known to cull females in anticipation of high mortality among males through hunting accidents, homicide, and suicide (Dickemann 1979: 341).”

Twins, being hard to nourish, were often discarded: “Mothers are unable to sustain two infants, especially where both are likely to be underweight. As Gray (1994: 73) notes, “even today, with the availability of western medical services it is difficult to maintain twins.” On Bali, which is otherwise extraordinary in its elevation of babies to very high esteem, giving birth to more than one child at a time is seen as evidence of incest. Priests consider the birth of twins as sub-human or animal-like (Lansing 1994; Barth 1993; Belo 1980). Similarly, the Papel (Guinea-Bissau) believe that it is mufunesa to give birth to many children at the same time like animals. Pigs have many offspring. Human beings give birth to only one each time. Therefore twins have to be thrown away. If not, the father, the mother, or somebody in the village may die. (Einarsdóttir 2004: 147).

Among the !Kung, Nancy Howell found that mothers whose toddlers had not been weaned might terminate the life of their newborn. In a society with high infant mortality (IM), an unweaned but otherwise thriving child is a better bet than a newcomer of unknown viability. The mother is expected by the band to kill one of a pair of twins or an infant with obvious defects. She would not be committing murder because, until the baby is named and formally presented in camp, it is not a person (Howell 1979: 120).

We can juxtapose this picture – paralleled in pre-modern communities the world over – with the almost legendary affection and love the !Kung show their young (Konner 2005). Similarly, Trobriand Island (Papua New Guinea) women, who also shower affection on their children, “were surprised that Western women do not have the right to kill an unwanted child … the child is not a social being yet, only a product manufactured by a woman inside her own body” (Montague 1985: 89).”

“In farming communities, additional farmhands are usually welcomed. Still, in rural Japan, a family would be subjected to considerable censure for having “too many” children and might find themselves ostracized if they failed “to get rid of the ‘surplus’” (Jolivet 1997: 118; see also Neel 1970). Bear in mind that breastfeeding is more costly – metabolically – than pregnancy (Hagen 1999: 331). In the impoverished northeast of Brazil, women can count on very little support from their child’s father, and their own resources are meager. Hence, “child death a mingua (accompanied by maternal indifference and neglect) is understood as an appropriate maternal response to a deficiency in the child. Part of learning how to mother … include[s] learning when to ‘let go’” (Scheper-Hughes 1987b: 190). Early cessation of nursing – one manifestation of the mother’s minimizing her investment – is supported by an elaborate folk wisdom that breast milk can be harmful, characterized as “dirty,”“bitter,”“salty,” or “infected.” Another folk illness category, doença de crianca, is used flexibly by mothers in justifying a decision to surrender the child into the hands of God or, alternatively, raise it as a real “fighter.” Of 686 pregnancies in a sample of 72 women, 251 infants failed to reach one year of age (Scheper-Hughes 1987a).”

“Long before the “one-child policy,” abortion was common in China. The oldest Chinese medical text found so far, some 5,000 years in age, includes reference to mercury as an abortifacient.”

I also hadn’t thought about traditional attitudes toward children with disabilities, or children (perhaps with autism) who don’t engage in eye contact, smiling, and other behavior that charms adults. “Hrdy (in press) suggests that the infant’s gaze-following and close attention to facial expressions and moods – along with a plump body and other neotenous features – are designed to send a clear signal to its mother and other caretakers: “Keep me!””

“In earlier times, the “difficult” or unwanted child might be dubbed a “changeling” or devil-inspired spirit, thereby providing a blanket of social acceptability to cloak its elimination (Haffter 1986). In cases where mothers are forced to rear unwanted children, the young may suffer abuse severe enough to end their life. While our society may treat such behavior by the parent as a heinous crime, “This capacity for selective removal in response to qualities both of offspring and of ecological and social environments may well be a significant part of the biobehavioral definition of Homo sapiens” (Dickeman 1975: 108).”

“Changelings represent a special sub-group of “demon” children who provoke a negative response from caretakers. The changeling was an enfant changé in France, a Wechselbag in Germany, and, in England, a “fairy child.” Strategies to reverse the switch included tormenting the infant or abandoning it in a lonely spot (Haffter 1986). A Beng mother-to-be who breaks a taboo may have her uterus invaded by a snake. The snake takes the fetus’s place and, after birth, is gradually revealed by the infant’s strange behavior. “The child may be harassed and hit by stones; however, being boneless like a snake, the snake-person is thought to feel no pain” (Gottlieb 1992: 145). A Papel infant deemed abnormal may be a spirit that’s entered the mother’s uterus. Two procedures are available to determine whether the child is human, but surviving either procedure seems improbable (Einarsdóttir 2008: 251). Dogon children thought to be evil spirits are taken: Out into the bush and you leave them … they turn into snakes and slither away … You go back the next day, and they aren’t there. Then you know for sure that they weren’t really [Dogon] children at all, but evil spirits. (Dettwyler 1994: 85–86) Among the Nuer, it is claimed, a disabled infant was interpreted as a hippopotamus that had mistakenly been born to human parents; the child would be returned to its proper home by being thrown into the river. (Scheer and Groce 1988: 28) In … northern Europe, changelings were left overnight in the forest. If the fairies refused to take it back, the changeling would die during the night – but since it was not human, no infanticide could have occurred. (Hrdy 1999: 465) [For Lurs] Djenn are said to be … jealous of the baby, especially during the first ten to forty days; they might steal the baby or exchange it for their own, sickly one. A baby indicates that it might be a changeling by fussiness, weakness, or lack of growth. (Friedl 1997: 69)”

Foragers vs. agriculturalists

Attitude toward children in general seems to vary by livelihood.

“In Central Africa, systematic comparisons have been drawn between foragers and farmers in the same region. Bofi-speaking foragers follow the !Kung model. Babies are carried or held constantly, by mothers and fathers, are soothed or nursed as soon as they cry, and may wean themselves after three to four years. Children are treated with the affection and respect consistent with preparing them to live in an egalitarian society where the principal subsistence strategy is cooperative net-hunting. Bofi-speaking farmers, on the other hand, tend not to respond as quickly to fussing and crying, are likely to pass the infant off to a slightly older sibling, and are verbally and physically abusive to children, who are treated like the farmhands they are soon to be.”

“The Garo, who live in the forests of Bengal, all share in infant and childcare, and parents “seldom roughhouse with their children, but play with them quietly, intimately, and fondly” (Burling 1963: 106). In the Northwest Territory of Canada, the Inuit (aka Eskimo) would never leave a child alone or let it cry for any length of time. Infants receive a great deal of solicitous care and lots of tactile comfort, anticipatory of “the interdependence and close interpersonal relations that are an integral part of Inuit life” (Condon 1987: 59; Sprott 2002: 54).

Draper observed a similar mindset operating among !Kung foragers in the Kalahar: Adults are completely tolerant of a child’s temper tantrums and of aggression directed by a child at an adult. I have seen a seven-year-old crying and furious, hurling sticks, nutshells, and eventually burning embers at her mother … Bau (the mother) put up her arm occasionally to ward off the thrown objects but carried on her conversation nonchalantly. (Draper 1978: 37)”

How children are expected to speak

“Clearly Euroamerican and Asian parents are preparing children to be more than merely competent native speakers. They encourage the development of narrative ability through frequent queries about the child’s activity, including their subjective assessments: “mothers pick up on children’s … topics, repeat and extend what their children say, and adjust their language … to support the child’s projects” (Martini 1995: 54). Toddlers are expected to hold and to voice their opinions! As parents seek “explanations” from their children, they also tolerate interruptions and contradiction (Portes et al. 1988). And this entire package of cultural routines is almost completely absent in the ethnographic record (Robinson 1988).

“In a Mayan community … children are taught to avoid challenging an adult with a display of greater knowledge by telling them something” (Rogoff 1990: 60). West African Wolof parents never quiz their kids by asking known-answer questions (Irvine 1978) – a favorite trick of Euroamerican parent-teachers. Fijian children are never encouraged to address adults or even to make eye contact. Rather their demeanor should express timidity and self-effacement (Toren 1990: 183).”

“Qualities we value, such as precocity, verbal fluency, independent and creative thought, personal expression, and ability to engage in repartee, would all be seen by villagers as defects to be curtailed as quickly as possible.25 These are danger signs of future waywardness. “Inquisitiveness by word or deed is severely censured, especially in [Kogi] women and children” (Reichel-Dolmatoff 1976: 283). “A [Sisala] child who tries to know more than his father is a ‘useless child’ (bichuola), for he has no respect” (Grindal 1972: 28). In rural Turkey the trait most valued by parents (60 percent) was obedience; least valued (18 percent) was independence (Kagitçibasi and Sunar 1992: 81).”

How do children learn?

“I discuss the prevailing view in WEIRD society – among most scholars as well as the public at large – that children’s development into mature, competent members of society depends critically on the guidance and lessons, beginning in infancy, provided by an eager parent who’s a “naturally gifted” teacher. Based on unequivocal evidence of the relative unimportance of teaching in the ethnographic record, I question that assumption as well as its evolutionary foundation.”

“De León (2012) records an episode from her Zinacantecan site where a three-year-old boy nearly runs, barefoot, through a fire. Adults do not react sympathetically. Instead, they comment that the child is flawed in not developing awareness of its surroundings, not paying close attention, and not figuring things out. There is an uneasy trade-off here. On the one hand, by indulging their curiosity about the environment and the things in it, parents insure that children are learning useful information without the necessity of parental intervention. This efficiency comes at a cost of the occasional damage to or loss of one’s offspring (Martini and Kirkpatrick 1992).”

“Active or direct teaching/instruction is rare in cultural transmission, and that when it occurs, it is not aimed at critical subsistence and survival skills – the area most obviously affected by natural selection – but, rather, at controlling and managing the child’s behavior.”

“Outside WEIRD or post-industrial society, this suite of parent–infant interaction patterns is rare. Mothers don’t often engage cognitively with infants, they may only respond contingently to their distress cues, and they probably do not gaze at them or engage in shared attention to novel objects (de León 2011: 100; Göncü et al. 2000; LeVine 2004: 161).”

In most traditional societies, children and young adults are expected to learn by observation rather than direct teaching.

In a Guatemalan indigenous community where people use a traditional learning style to approach factory work: “The newly hired worker performs menial tasks33 such as bringing material to the machine or taking finished goods off of it, but most of the time is spent observing the operations of the person running the machine. [The new worker] neither asked questions nor was given advice. When the machine snagged or stopped, she would look carefully to see what the operator did to get it back into motion … This constituted her daily routine for nearly six weeks, and at the end of this time she announced that she was ready to run a loom … and she operated it, not quite as rapidly as the girl who had just left it, but with skill and assurance … at no time during her learning and apprentice period had she touched a machine or practiced operating … She observes and internally rehearses the set of operations until she feels able to perform. She will not try her hand until she feels competent, for to fumble and make mistakes is a cause for verguenza – public shame. She does not ask questions because that would annoy the person teaching her, and they might also think she is stupid. (Nash 1958: 26–27)”

“I provide an extended example, of mother Sua and daughter Nyenpu each weaving a fishnet. As the vignette unfolded, the main point seemed to be how little interest Sua had in getting involved in Nyenpu’s weaving. Sua claimed that her stance was typical and replicated her own mother’s attitude when she was learning net-weaving. Several other informants told me of approaching experts for help and being rebuffed (Lancy 1996: 149–150). Other ethnographers report similar tales. Reichard describes a Navajo girl who learned to weave in spite of her mother’s repulsing her interest (1934: 38), which paralleled a case from Truk of a weaver/basket-maker whose kin were unsupportive of her efforts to learn their skills (Gladwin and Sarason 1953: 414–415), and a case from the Venda tribe where a potter is vehement that “‘We don’t teach. When women make pots some (children and others) come to watch, then go and try’” (Krause 1985: 95).”

A Javanese shellfish diver responds to the question of whether she learned the practice from her mother:

“My mother! she said loudly, She drove me away! I tried to follow her to the bottom to watch, but she shoved me back. When we were on the surface again, she practically screamed at me to move OFF and find my danged abalone BY MYSELF. So we had to discard [one] cliché about how artisans learn. (Hill and Plath 1998: 212)”

There are a few cases of explicit teaching:

“There are a few cases in the literature of grandmothers conducting educational tours through the bush to acquaint their younger kin with medicinal plants (Ngandu – Hewlett 2013: 76; Tonga – Reynolds 1996: 7).”

“An interesting “work-around” for the prohibition on teaching is provided by the Fort Norman Slave [Canada], who hunt during severe winter weather and must traverse ice-fields. Fathers “instruct” sons about this dangerous environment (which comprises thirteen kinds of ice and multiple modes of travel) via a game-like quiz (Basso 1972: 40).”

Analytic thinking

While there's a lot of knowledge being transmitted in traditional societies, like how to make and use a blowgun for hunting or how to hollow a canoe, analysis and taxonomy seem to be absent in societies where people haven't gone to school. Lancy cites Alexander Luria's 1930s interviews with peasants in Central Asia:

"In the first example we can see the villager reasoning from personal experience (or lack thereof) and inability or unwillingness to apply a general rule. 'Problem posed: 'In the Far North, where there is snow, all bears are white. Novaya Zemlya is in the far north and there is always snow there. What color are the bears?' Response: 'We always speak of only what we see; we don’t talk of what we haven’t seen.' (Luria 1976: 108)

In another problem, men and women were asked to sort and group various kinds and colors of weaving yarn (Uzbekistan is noted for its carpets). The male response was 'men [not being weavers] don’t know colors and call them all blue.' The women refused to impose any grouping or organization – something educated Uzbeks did quite easily – exclaiming that 'none of these are the same' (Luria 1976: 25, 27).

In a fishing community in Sulawesi, Vermonden found directly parallel results, with fishers resistant to discussing marine life more generally; they eschewed speaking of types of fish or of considering different ways of grouping them. Their thinking was governed by their practice (true also for Penan hunters – Puri 2005: 280 – and South American and African subsistence farmers – Henrich et al. 2010: 72). . . Had Vermonden’s informants been schooled, they might have used broader and more inclusive organizing principles and been able to display a more encyclopedic knowledge of fish." I assume that these people did in fact know a lot about fish, yarn, etc, which were their daily livelihood, but were used to thinking in practical terms.

(I was telling Jeff the bear example at dinnertime. "It's white," piped up our four-year-old without prompting. She's used to "known-answer questions" where grownups ask you things even though they know the answer.)

Learning to be street-smart

While children in some societies need to learn to avoid predators and poisonous plants, Lancy also briefly covers urban environments where children must be equipped for other dangers. "A mother in a favela of Rio de Janeiro knows “intuitively that in order for her children to survive, toughness, obedience, subservience, and street smarts are necessary; otherwise, the child can end up dead” (D. Goldstein 1998: 395)."

Charles Dickens depicts a similar strategy in 19th-century London, with a father describing how he's trained his son: “I took a good deal o’ pains with his eddication, sir; let him run in the streets when he was very young, and shift for hisself. It’s the only way to make a boy sharp, sir” (Dickens 1836/1964: 306)."

Learning through play

“Play is a truly universal trait of childhood. The one thing that children can appropriate for themselves, without the sanction of culture or explicit blessing of parents, is play. It is ubiquitous. A baby will play with its mother’s breast. The first glimmer of understanding about the natural world and how it works comes through play with objects. After its nurturing mother, the child’s first close relationships are with its playmates – usually siblings. The child’s first active engagements with the tasks that will occupy most of its adult life – hunting, cooking, house-building, baby-tending – all occur during make-believe.”

“Many of the child’s most basic needs seem to be fed by play – their need to socialize with peers and their need for physical, sensory, and, to a lesser extent, cognitive stimulation (Lancy 1980a). The demands of earning a living and reproduction gradually extinguish the desire to play. This happens earlier in girls than in boys – almost universally.”

Modern children:

“To encourage object play, we provide lots of toys, including safe, miniature tools, in various sizes, along with the dolls to use them. We also provide objects to play with that are specifically designed to facilitate the kind of cognitive complexity and flexibility that many assert is the raison d’être of object play (Power 2000). And, what is perhaps most remarkable, we sometimes intervene to “teach” our children how to use their toys or nudge them into more complex uses (Gaskins et al. 2007). I have found only one example of this in the ethnographic literature – a Wogeo father assisting his son with a miniature canoe (Hogbin 1946: 282) – and I am confident it occurs rarely. In research where the investigators created conditions designed to facilitate their involvement, East Indian and Guatemalan villagers would not intervene in their toddlers’ play (Göncü et al. 2000). It’s hard to escape the conclusion that our “micro-management” of children’s toys and play is driven by the inexorable demands of schooling.”

In contrast to play with specially provided objects, social play and pretend play are ubiquitous.

“Comparing across fifteen species of primates, observers found a statistically reliable relationship between cerebellum size and time devoted to social play (Lewis and Barton 2004; see also Fisher 1992).”

“This rapid growth in understanding – correlated with a rapidly growing brain – emerges in early childhood as two powerful motives. These are, first, to “fit in,” to be liked, appreciated, and accepted. The second motive force is a drive to become competent, to replicate the routine behaviors enacted by those who’re older and more capable. The presence of these drives accounts for the child’s ability to learn through observation, imitation, and, by extension, playing with objects and ideas in make-believe.”

“Esther Goody describes the richness and complexity of make-believe cooking in a village in north Ghana. Miniature kitchens are constructed, ingredients gathered, and soup made, all the while accompanied by singing and the construction of play scripts that mimic adult discourse. And, of course, the girls must insure that their play enfolds the younger siblings who are in their care. Boys have bit parts in these playlets as “husbands,” and are limited to commenting on the flavor of the soup (Goody 1992).

My kids and their cousins are avid mud-soup makers. Boys are full participants in this case

“Make-believe reveals children’s insight into the adult world. Araucania boys accurately mimic the speech and movements of drunken males celebrating fiesta (Hilger 1958: 106). Yanamamo boys pretend to “smoke” hallucinogens and then stagger around in perfect imitation of their stoned fathers acting as shamans (Asch and Chagnon 1974).”

Of course, an anthropologist in the village provides an interesting topic for pretend play: “Parenthetically, many an anthropologist has seen herself or himself reflected (unflatteringly) in the play of erstwhile subjects (Bascom 1969: 58).”

“The doll is arguably the most widely found toy and the range of materials used and designs employed is immense (Ruddle and Chesterfield 1977: 36).16 From rags tied into a shapeless bundle to high-tech baby dolls that produce a babble of baby-talk, wet themselves, and eagerly move their limbs, the variety is fascinating. While baby dolls seemed to have been a universal adjunct to Roman girls’ play, lower-class girls had infant dolls that they mock-nursed, comforted, and cleansed while upper-class girls, whose future as adults would not include childcare, dressed and primped the ancient equivalent of Barbie (Wiedemann 1989: 149–150).

Not all cultures encourage play

“Play may be seen as a sign of waywardness. Bulusu’ view play as naughty (jayil) and those who play “too much” as crazy (mabap) (Appell-Warren 1987: 160). Children may be scolded for getting dirty or telling stories they know aren’t true (e.g. fantasizing) (Gaskins et al. 2007: 192). On Malaita Island, where children are expected to carefully observe and report on newsworthy events in the village, children’s fantasy constructions are discouraged; they “are mildly reprimanded with ‘you lie’” (Watson-Gegeo and Gegeo 2001: 5).

Following the Protestant Reformation, many influential authorities condemned play in general as well as specific kinds of play, such as solitary play or contact sports. Morality came to be equated with decorum and emotional restraint; “indulging children was a cardinal sin” (Colón with Colón 2001: 284).

Similar sentiments were expressed by Chinese sages: Huo T’ao had no tolerance for play … as soon as a child is able to walk and talk, it must be taught not to play with other children. Children must practice treating one another as adults … When [children] see each other in the morning, they must be taught to bow solemnly to each other. (Dardess 1991: 76)”

When do parents play with young children?

Parents playing with babies varies a lot:

“An analysis of 186 archived ethnographies of traditional societies indicated wide variation in the amount of mother-infant play and display of affection (Barry and Paxson 1971). In a more recent comparative observational study, “Euro-American adults were much more likely than Aka or Ngandu adults to stimulate (e.g., tickle) and vocalize to their infants. As a result, Euro-American infants were significantly more likely than Aka and Ngandu infants to smile, look at, and vocalize to their care providers” (Hewlett et al. 2000: 164).21 Play with infants also seems generally less common among agrarian societies; for example, an Apache (North American agro-pastoralists) “mother sometimes plays with her baby … A father is not likely to play with a baby” (Goodwin and Goodwin 1942: 448). In hundreds of hours of close observation of parent–child interaction among Kipsigis (Kenyan) farmers, Harkness and Super (1986: 102) recorded “no instances of mothers playing with their children.””

“Among the !Kung, parents not only don’t play with their children post-infancy, they reject the notion outright as potentially harmful to the child’s development. They believe that children learn best without adult intervention (Bakeman et al. 1990: 796). The mother of a toddler not only faces potential conflict between childcare and work, she’s likely pregnant as well. I would argue that the mother’s greatest ally, at this point in the childrearing process, is the magnetic attraction of the sibling or neighborhood play group (Parin 1963: 48). The last thing a pregnant mother wants is for her child to see her as an attractive play partner. Even verbal play is avoided.”

I found this such a relief to read. The hardest stage of parenting for me was caring for an infant while being my two-year-old’s only regular playmate. She had an insatiable desire for stories, and I just wasn’t up for it.

Young monkeys play with each other, but chimpanzee mothers play with and tickle their babies.

“Why is the chimpanzee mother providing her baby with what monkey infants get from their peers? One clue in the direction of an answer may be the group structure of chimpanzees. I observed that chimpanzee mothers spend most of their time alone with their babies. As a consequence it is the chimpanzee mother who has to give her baby this sort of interaction if he gets it at all. (Plooij 1979: 237) "

Similar forces may promote mother–child play among humans. The small band of “Utkuhikhalingmiut [Inuit are] the sole inhabitants of an area 35,000 or more miles square” (Briggs 1970: 1). Aside from the almost total lack of other children to play with, the mother–child pair is isolated inside their igloo for days on end during the worst weather. Jean Briggs observed mothers talking to their children, making toys for them, playing with them, and encouraging their language development. Further, there is every reason to believe that modern living conditions in which infants and toddlers are isolated from peers in single-parent or nuclear households produce a parallel effect. That is, like chimps in the wild, modern, urban youngsters only have access to their mothers as potential play partners. In Japan, the mother–child pair has become quite isolated, sequestered in high-rise apartment buildings.”

This sounds very familiar to me.

Learning through chores

Learning to do the chores of adult daily life is of great interest to children everywhere.

“In the Giriama language the term for a child roughly two through three years in age is kahoho kuhuma madzi: a youngster who can be sent to fetch a cup of water … A girl, from about eight years until approximately puberty, is muhoho wa kubunda, a child who pounds maize; a boy of this age is a muhoho murisa, a child who herds. (Wenger 1989: 98)”

“Generally speaking, a girl’s working sphere coincides with that of her mother: the household, kitchen, nursery, laundry, garden, and market stall. (Paradise and Rogoff 2009: 113) depict a five-year-old Mazahua girl closely following her mother’s lead in setting up an onion stand in the market – trimming, bunching, and arranging their onions. When invited to establish a satellite onion stand, 'her excitement is unmistakable and she quickly takes the initiative in finding an appropriate spot and setting it up.'”

“In WEIRD society, parents and adults generally take every opportunity to instruct children, even when they are patently unmotivated or too awkward and immature. The term “scaffolding” may be used to describe the process whereby the would-be teacher provides significant assistance and support so that the novice can complete a task that is otherwise well beyond his grasp (McNaughton 1996: 178). Elaborate scaffolding is rarely seen elsewhere (Chapter 5). No one wants to waste time teaching novices who might well learn in time without instruction.”

“Little girls strap bundles of leaves on their backs as babies, boys build little houses … A little girl accompanying her mother to the fields practices swinging a hoe and learns to pull weeds or pick greens while playing about … Playing with a small gourd, a child learns to balance it on his head, and is applauded when he goes to the watering-place with the other children and brings it back with a little water in it. As he learns, he carries an increasing load, and gradually the play activity turns into a general contribution to the household water supply. (Edel 1957/1996: 177)”

“In the Sepik region of Papua New Guiena, Kwoma children eagerly embrace the piglets they’re given to protect, raise, and train (Whiting 1941: 47). Talensi boys are said to possess “a passionate desire to own a hen” (Fortes 1938/1970: 20).”

“The Touareg boy progresses from a single kid (at three years of age) to a herd of goats (at ten) to a baby camel (at ten) to a herd of camels (at fifteen) to managing a caravan on a trek across the Sahara (at twenty). Preferentially, the aspirant herder interacts with and learns from herders who are slightly older, not adults. Adults are too forbidding to ask questions of or display ignorance in front of. Above all, it is a hands-on experience, as “The abstract explanation so typical of our schooling is completely absent” (Spittler 1998: 247).”

“Four-year-old Bafin has already grasped the meaning of sowing and is able to perform the various movements … he is entrusted with an old hoe as well as with some seeds so that he can gain some practice in this activity. However … he has to be allocated a certain part of the field where he neither gets in the way of the others nor spoils the rows they have already sown … As a rule, his rows have to be re-done. (Polak 2003: 126, 129)”

This is one of the few passages that got at my concern about children’s involvement in chores: it usually creates more work for the parents. It did persuade me to let Anna load the dishwasher, which she does ineptly but avidly.

Chores vs. crafts

“I was surprised to discover that, in Gbarngasuakwelle, there is a gulf between the chore curriculum and what we might call the craft curriculum. The former is often compulsory – a child may be severely chastised or beaten for failure to complete appropriate chores satisfactorily. The latter is not only entirely voluntary, but children seem to be offered little encouragement in it. Indeed, they may be actively discouraged from trying to learn a craft or otherwise complex trade.”

“Somewhat later, the child may elect to move beyond the core skills expected of everyone to tackle more challenging endeavors such as learning pottery or weaving. She or he must demonstrate adequate strength, physical skill, and motivation before anyone will deign to spend time on his or her instruction.”

Rites of passage

Most traditional societies involve some initiation ceremony to mark the transition to adulthood, may involving “days of hazing, fasting, beating, sleeplessness, and sudden surprises.”

After being raised by women, boys’ rituals often focus on separating them from the world of women:

“One element that looms large in the training of male adolescents in much of Africa and Papua New Guinea is misogyny, as noted above. There is a distinct focus on teaching boys to feel superior toward and contemptuous of women. The “text” of many messages conveyed to initiates is replete with references to women’s physical weakness relative to men and their power to pollute through menstrual and puerperal blood. Another tool in the men’s arsenal is the use of “secrets,” including sacred terms, rituals, locations, and objects such as masks. These “secrets” are denied to women on pain of death. For the Arapesh (Sepik Region), “initiation ceremonies [include] an ordeal followed by the novices being shown the secret paraphernalia … flutes, frims, paintings, statues, bullroarers” (Tuzin 1980: 26). Denying female access to powerful spirit forces aids in maintaining male hegemony. A Mehinacu girl “cannot learn the basic myths because the words ‘will not stay in her stomach’” (Gregor 1990: 484). Wagenia “women and girls belong to the social category of the non-initiated, from whom the secrets of initiation were carefully concealed” (Droogers 1980: 78).”

“Immediately following [the ordeal], the initiators drop their razors, spears, cudgels or what have you, and comfort the boys with lavish displays of tender emotion. What resentment the latter may have been harboring instantly dissipates, replaced by a palpable warmth and affection for the men who, moments before, had been seemingly bent on their destruction. As their confidence recovers itself, the novices become giddy with the realization that they have surmounted the ordeal. (Tuzin 1980: 78)”

“The Hitler Youth and the Soviet Young Pioneers both capitalized on the idealism and fanaticism characteristic of adolescence (Valsiner 2000: 295; see also Kratz 1990: 456). During the Cultural Revolution, Chinese authorities used the naturally “anti-social,” rebellious nature of adolescents in recruiting, training, and then setting them loose as “Red Guards” to destroy bourgeois, Western, or intellectual elements of Chinese society (Lupher 1995). Today, Muslim terrorist organizations easily recruit male and female adolescents to serve as suicide bombers. Again, there are fundamental biological and psychological aspects of adolescence that render them susceptible to group-think mentality. Normal standards of human decency are suspended, allowing them to commit crimes in the name of the group.”

Neither here nor there

The author describes the plight of young people who have been socialized away from their traditional cultures but not given anything good in exchange:

“Christian missions offer them the opportunity to escape the restrictions imposed by traditional rites associated, in the Sepik area, with the men’s Haus Tambaran, without successfully socializing them to embrace Western/Christian values. Similarly, in attending government schools, young males signal their abandonment of the traditional agrarian economy without actually learning enough to secure a job in the modern economy. In short, they have been led to believe they are superior to the senior men, yet bring no significant resources to the community”

“Disaffected African students, their hopes for white-collar jobs dashed by stagnant economies, are easily recruited as “rebels” (Lancy 1996: 198) and street rioters (Durham 2008: 173). Terrorists and rebel armies capitalize on the peculiarities of adolescent psychology, brought on in part by “living in limbo,” to create pliable fanatics (Rosen 2005: 157). Rosen also notes the continuity between traditional Mende warrior training, described earlier in this chapter, and the recruitment and training of child soldiers.”

The decades-long Salvadoran Civil War raised a generation of men with no livelihood other than war:

“Initiation rites in the socialization of young rebels, unlike traditional rites, do “not facilitate their social transition into responsible adulthood” (Honwana 2006: 63). Similarly, in the Salvadorian civil war, young soldiers “were not given a chance to practice and learn how to be campesinos, dedicated to subsistence agriculture … and the lack of preparation for a new, adult peacetime identity led many youth to choose the negative identity of … marero [delinquent/gang member]. (Dickson-Gómez 2003: 344–345)”

“Similarly, adolescent males living on Indian reservations suffer mortality and suicide rates three times the national average.”

Traditional cultures meet Western schools

Western schools were historically places where knowledge is crammed and beaten into children.

4000 years ago a Sumerian student described his day: “My headmaster read my tablet, said: ‘There is something missing,’ caned me. ‘Why didn’t you speak Sumerian,’ caned me. My teacher said: ‘Your hand is unsatisfactory,’ caned me.’ And so I began to hate the scribal art” (Kramer 1963: 238–239).”

“Until fairly the 1970s, elite English boarding schools (and their US counterparts) for males weren’t all that different in terms of the constant hazing of younger by older boys, the emphasis on physical deprivation and removal from family, and daily engagement in team sports. This is probably what prompted Arthur Wellesley, the duke of Wellington, to remark: 'The battle of Waterloo was won on the playing-fields of Eton.'"

“The lamentations of passionate critics provide another window on the nature of schooling. These critics believed that the reluctant scholar problem could be solved by making schooling more like the experiences of the unschooled child, mixing in play, letting the child make choices, rewarding curiosity and independent learning. The fact that these pleas continue to appear over nearly two millennia suggests how enduring and intractable were the earliest ideas about the nature of schooling.”

“The idea that school should interest children was considered a radical new pedagogical philosophy in the United States of the 1840s”

But although the West has moved into a more child-centered mode, schools in the developing world remain on an old-fashioned model:

“As schools are introduced to formerly school-less communities, they much more closely resemble medieval schools than they do modern, progressive institutions. Bare, drafty classrooms, rote memorization, a scarcity of teaching materials, corporal punishment, unintelligible teachers, menial labor by students, the underrepresentation and exploitation of girls – all harken back to the dawn of schooling in the West.”

“Schools have encountered resistance from pupils who struggle to “sit still” or to meet the teacher’s gaze; from parents who’d prefer their children to be working and who reject their assigned role as “under-teacher,” prepping and supporting their child’s schooling; from patriarchal societies that impose limits on the choices available to women; and from the general public because of the very poor quality of instruction and the coercive atmosphere.”

“In a survey of childhood across history and culture, the suite of practices and teaching/learning abilities associated with modern schooling is largely absent”

An anthropologist “marvels at how facile and active the Matses children are in the natural environment, compared to what she feels is her own ineptitude. She is cowed by three- and four-year-olds who competently paddle and maneuver canoes on the wide river. She observes young boys nimbly catching and handling enormous catfish (Figure 28). And then she is struck by the painful contrast between the children’s mastery of their natural surroundings and the great discomfort and incompetence they display in the classroom. She summarizes the dilemma as 'learning to sit still.'"

The demographic transition

400 years ago, a change began to happen in the Netherlands:

“In the seventeenth century, foreigners were already recording their astonishment at the laxity of Dutch parents … they preferred to close their eyes to the faults of their children, and they refused to use corporal punishment … foreigners remarked on something else: since the sixteenth century, most Dutch children – girls as well as boys – had been going to school. (Kloek 2003: 53)”

“John Locke – exiled to Holland in 1685–1688 – was profoundly influenced by what he saw. His treatise on childrearing, published in 1693, brought Dutch ideas on childcare to England (Locke 1693/1994). At the end of the eighteenth century, the Quakers also embraced population control and used various means to reduce their fertility. “The drop in the birth rate also reflected … a rejection of the view that women were chattels who should devote their adult lives to an endless cycle of pregnancy and childbirth” (Mintz 2004: 78).”

Dutch paintings of this era are no longer only stiff portraits, but depict families enjoying time together (though I'm not sure how much the cat is enjoying this experience.)

"Teaching a cat to read", Jan Steen, 1660s. Note the young teacher holds a switch - this is how lessons work even in the relaxed Netherlands.

In developing countries, traditional methods of spacing births may be discouraged, resulting in a baby boom:

“For example, from Malaita Island in the South Pacific, traditional Kwara’ae practice was to keep men separated from their nursing wives for at least a year. However, the “abolition of the tabu system and the ascendance of Christianity has meant that … ritual separation [is] no longer practiced” (Gegeo and Watson-Gegeo 1985: 240–241). As a result, fertility has jumped and families with ten to thirteen children are not uncommon.”

Western intervention has addressed one aspect of population but not another: “The agencies that intervened to reduce infant mortality were not as ready with contraception and family-planning interventions, and the result has been masses of humanity living on the ragged edge of poverty.”

Even where it's available, people may not be interested in birth control, despite the practical difficulties of raising lots of children. In Burkina Faso:

"There are no perceived disadvantages in having lots of children. Children are never seen as a drain on resources. The availability of food is believed to be purely a product of the God-given fortune of the child, and nothing to do with the level of resources available within the household or the number of mouths to feed [because] 'every child is born with its own luck.' (Hampshire 2001: 115)"

The author gets editorial at times, quipping "The rich get richer, and the poor get lots of sickly children."

“Unfortunately, the ubiquity of infant death along with well-established coping mechanisms inures people to a phenomenon that, given the state of medical knowledge and a pharmacopeia adequate to the task, shouldn’t be happening. The wastage of young human life and the debilitating impact this has on mothers are staggering and cannot possibly be justified. And, in the West, we remain largely oblivious of the problem of child malnutrition and death in the Third World until it reaches such proportions that the story becomes newsworthy.”



Discuss

Структурирование

События в Кочерге - 27 августа, 2020 - 19:00
Структурирование — формат, на котором участники выдвигают вопросы или темы, на которые они хотели бы «поструктурироваться», то есть эксплицитно разобраться с моделью проблемы, которая у них уже есть, и построить вместе с напарником что-то новое поверх; собираются в пары по интересу к предложенным темам, и говорят один на один.

Preface to the sequence on economic growth

Новости LessWrong.com - 27 августа, 2020 - 18:37
Published on August 27, 2020 3:37 PM GMT

On Lesswrong, when we talk about artificial intelligence, we tend to focus on the technical aspects, such as potential designs, specific developments, and future capabilities. From an engineering perspective, this focus makes sense. But most people here aren't interested in artificial intelligence because they want to know how AI will be designed; the reason we're here is because AI has the potential to radically reshape the world around us.

Longtermists have often emphasized the role economic growth plays as perhaps the most important phenomena of human history. In a quite real sense, economic growth is what distinguishes 21st century humanity from our distant ancestors who had no technology or civilization. Nick Bostrom summarizes this point well,

You could argue that if we look back over history, there have really only been two events that have fundamentally changed the human condition, the first being the Agricultural Revolution some 10,000 or 12,000 years ago in Mesopotamia, where we transitioned from being hunter-gatherers, small bands roaming around, to settling into cities, growing, domesticating crops and animals. [...]The second fundamental change in the human condition, Industrial Revolution, where for the first time, you have the rate of economic and technological growth outstripping population growth, and so only when this happens can you have an increase in average income. Before that, there was technological growth and economic growth, but the economy grew 10%, the population grew 10%, everybody's still in a Malthusian condition.

Many theorists anticipate that there will be a third fundamental change in the human condition, roughly timed with the development of advanced artificial intelligence. In line with these predictions, economic growth is the primary specific benchmark people have used to characterize potential future AI takeoff.

If economic growth is the essential variable we should pay most attention to when it comes to AI, then our understanding of AI takeoff will be woefully incomplete without a grasp of what drives economic growth in the first place. To help mitigate this issue, in this sequence I will explore the underpinnings of modern economic growth theory, and then try to relate economic theory to AI developments. In doing so, I aim to identify crucial pieces of information that may help answer questions like,

  • How much technological progress in the past has been bottlenecked by investment as compared to insights?
  • How soon after advanced AI is created and turned on should we expect rapid economic progress to follow? Is there typically a large lag between when technologies are first demonstrated and when they heavily impact the economy?
  • What are the key factors for why AI is different from other technologies in its ability to induce rapid growth? Is it even different at all?

To provide one specific example of how we can import insights from economic growth theory into our understanding of AI, consider the phenomenon of wealth inequality between nations in the world. Wealth inequality between nations is ultimately the result of historical economic growth inequality, but things weren't always so unequal. Before the industrial revolution, per-capita wealth was approximately equal for all civilizations--at subsistence level. This state of affairs only changed when economic growth began to outstrip population growth in some nations during the industrial revolution.

AI takeoff can also be described in terms of growth inequality. A local (foom) intelligence explosion could be defined as an extremely uneven distribution of economic growth following the creation of superintelligent AI. A global (multipolar) takeoff could therefore be defined as the negation of a local intelligence explosion, where economic growth is distributed more evenly across projects, nations, or people.

Before we answer the important question of which version of AI takeoff is more likely, it’s worth recognizing why historically, growth inequality began after the industrial revolution. The factors that drove growth in the past are likely the best keys for understanding what will drive it in the future.

Organization of the sequence

Below, I have included a rough sketch of this sequence. It is organized into three parts.

The first part will provide the basic mechanics behind models of economic growth, and some standard results, with an emphasis on the factors driving technological innovation. Upon some research, and a recommendation from Alex Tabarrok’s blog, I have chosen to summarize the first several chapters of The Economics of Growth by Philippe Aghion and Peter Howitt.

The second part will dive into a recently developed economic model under the name Unified Growth Theory which the creator Oded Galor claims is the first major attempt to model the deep underlying factors driving economic growth throughout human history, cohesively explaining the onset of the industrial revolution and the emergence of the modern growth era. To provide some credibility here, the book introducing the theory has been reviewed favorably by top growth researchers, and Oded Galor is the editor in chief of the Journal of Economic Growth.

The third part will connect economic growth theory to artificial intelligence. Little research has been done so far examining the key economic assumptions behind the AI takeoff hypothesis, and thus it is possible to get a comprehensive survey of the published work so far. I will review and summarize the main papers, hopefully distilling the main insights generated thus far into a few coherent thoughts.

Other ways economic growth is relevant

Besides being a fixture of how people characterize AI takeoff, economic growth is potentially important for effective altruists of all backgrounds. For instance, in an effective altruism forum post, John Halstead and Hauke Hillebrandt argue that effective altruists have given short shrift to evidence that the best way to reduce poverty is to spur economic growth, rather than to distribute medicine or cash directly.

Economists have characterized the impacts of climate change primarily by its effects on growth, which has important implications for how much we should prioritize it in our longtermist portfolio. Similar statements can be made about the relative priority of pandemics, recessions, and in general a wide variety of global issues.

Economic growth is also just a critical piece of the human story. Without a basic understanding of growth, one's understanding of history is arguably horrible. From Luke Muehlhauser,

Basically, if I help myself to the common (but certainly debatable) assumption that “the industrial revolution” is the primary cause of the dramatic trajectory change in human welfare around 1800-1870 then my one-sentence summary of recorded human history is this:"Everything was awful for a very long time, and then the industrial revolution happened."Interestingly, this is not the impression of history I got from the world history books I read in school. Those books tended to go on at length about the transformative impact of the wheel or writing or money or cavalry, or the conquering of this society by that other society, or the rise of this or that religion, or the disintegration of the Western Roman Empire, or the Black Death, or the Protestant Reformation, or the Scientific Revolution.But they could have ended each of those chapters by saying “Despite these developments, global human well-being remained roughly the same as it had been for millennia, by every measure we have access to.” And then when you got to the chapter on the industrial revolution, these books could’ve said: “Finally, for the first time in recorded history, the trajectory of human well-being changed completely, and this change dwarfed the magnitude of all previous fluctuations in human well-being.”

Discuss

Covid 8/27: The Fall of the CDC

Новости LessWrong.com - 27 августа, 2020 - 17:10
Published on August 27, 2020 2:10 PM GMT

Most weeks, the disaster that was the head of the FDA not having any understanding of statistics and not making any attempt to think about the world would have been the headline. Then the CDC decided to revise its guidelines on testing from being for it to largely being against it, under pressure from the White House, and suddenly it’s the B-story.

It seems that every day there is a new thing surfacing to enrage me. The difference is that early in the pandemic, every day something would terrify me. I’m still periodically scared in an existential or civilization-is-collapsing-in-general kind of way, but not in a ‘the economy is about to collapse’ or ‘millions of Americans are about to die’ kind of way. 

I’m not sure whether this is progress.

Either way, a reminder that I’ve started a sports, sports gambling and sports modeling substack to avoid cluttering up this blog, so check it out here if you have yet to do so and that is relevant to your interests.

That’s out of the way. Let’s run the numbers.

Positive Test Counts DateWESTMIDWESTSOUTHNORTHEASTJune 11-June 1741976225107578717891June 18-June 24662922679210722115446June 25-July 1857613497416347216303July 2-July 81038794013920286318226July 9-July 151083955322925007220276July 16-July 221175065779726522120917July 23-July 291102196790324066726008July 30-Aug 5910026446221294523784Aug 6-Aug 12930426193118848621569Aug 13-Aug 19808876338415699820857Aug 20-Aug 26675456654013232218707

Apologies for the lack of charts last week – it seems they didn’t copy properly from Google docs to WordPress and I didn’t notice. 

Positive test counts are declining rapidly in the West and South, and slowly in the Northeast. Before, I would have considered that strong evidence that things are going great in those areas. Now, with testing on the decline, it’s not that simple, and we’ll have to dig into the positive test percentages and deaths to know for sure.

Deaths DateWESTMIDWESTSOUTHNORTHEASTJune 11-June 17778104012071495June 18-June 2483185912041061June 25-July 18586581285818July 2-July 88945591503761July 9-July 1513805392278650July 16-July 2214696743106524July 23-July 2917077004443568July 30-Aug 518317194379365Aug 6-Aug 1217386634554453Aug 13-Aug 1915768504264422Aug 20-Aug 2615037453876375

Deaths continue to tell the story that the South and West have turned the corner, the Northeast is still making steady progress, and the Midwest did indeed have deaths transferred from the Aug 6 – Aug 12 week to the Aug 13 – Aug 19 week and is still steady or slowly getting worse as of a few weeks ago. 

Each day, we see the 7-day average deaths fall slightly. If anything, it’s a little too consistent and makes me suspicious, but that is probably paranoia. Probably.

Positive Test Percentages By Region

Positive test percentages are actually up slightly in the South this week, down in the Northeast, and down slightly in the West:

PercentagesNortheastMidwestSouthWest7/16 to 7/222.49%5.13%13.29%8.56%7/23 to 7/292.54%5.51%12.32%7.99%7/30 to 8/52.58%7.26%12.35%6.68%8/6 to 8/132.30%5.67%14.67%6.98%8/13 to 8/202.06%5.62%9.41%6.47%8/20 to 8/261.86%5.78%9.93%5.88%

Hospitalizations are slightly down as well.

A decrease in testing should slightly increase positive test percentages, so this isn’t inconsistent with the South’s situation continuing to improve, but it’s also definitely not a good sign. A reminder that you can check the state-by-state data in my spreadsheet under the Positive Test Percentages tab here

For the first time, the Midwest did more tests this week than the West did, and they continue to have similar positive percentages on tests, so it will be interesting to see how long it takes their death differential to close.

Hawaii seems to have turned a corner, or at least it’s no longer seeing unchecked exponential growth. Arizona and Florida continue to make steady progress, with Arizona continuing to go faster. Texas news is not as good, and looks ambiguous due to its dramatic drop in testing. 

South Dakota and Iowa saw big scary jumps in positive test percentages. North Dakota also got noticeably worse. Minnesota continues to be headed in the wrong direction and looks poised for a new wave of protests and unrest.

Test Counts DateUSA testsPositive %NY testsPositive %Cumulative PositivesJune 11-June 173,453,4404.6%442,9511.1%0.66%June 18-June 243,686,3365.9%440,8331.0%0.72%June 25-July 14,352,9817.1%419,6961.2%0.82%July 2-July 84,468,8508.2%429,8041.1%0.93%July 9-July 155,209,2438.4%447,0731.1%1.06%July 16-July 225,456,1688.6%450,1151.1%1.20%July 17-July 295,746,0567.9%448,1821.1%1.34%July 30-Aug 55,107,7397.8%479,6131.0%1.46%Aug 6-Aug 125,121,0117.3%502,0460.9%1.58%Aug 13-Aug 195,293,5366.2%543,9220.8%1.68%Aug 20-Aug 264,785,0566.0%549,2320.8%1.77%

After two weeks of what looked like stable test counts, we once again headed substantially in the wrong direction. The positive test percentage declining in spite of this is of course good news, but raises the worry of whether testing has shifted to where it is least needed while being shut down where it is needed most. New York’s testing keeps expanding even as its case counts decline. 

The numbers look like one would have expected them to look last week, plus a decline in tests. There is slow improvement in deaths and hospitalizations. The outlook continues to be that things will continue much as they are for quite a while, in terms of the virus itself.

Thus, the news this week lies elsewhere.

Head of FDA Fails Statistics Forever

See this tweet from my friend Andrew Rettek, noting that the head of the FDA is far from alone in failing statistics forever. 

(Or as the new cowards in charge of the page at TVTropes are now calling it, ‘Artistic License: Statistics.’ To me, that’s another sign of an alarming societal shift. It’s not ‘artistic license,’ it’s some combination of lying/fraud and a failure to understand how numbers work on a deep and essential level. There are times when it makes sense to use a little ‘artistic license’ with your physics or biology to tell a good story. But much more frequently, They Just Didn’t Care about what is true and what causes what in the physical world, the same as others who are making actual decisions with actual consequences don’t care either. If you don’t apply for the license, you don’t have one. Period. I call for the immediate and total restoration of failing forever.)

And there isn’t one for statistics or mathematics. You just fail. Period.

The mistake here isn’t one that someone who knows how numbers work would ever make. It’s also not one that people who think about what their statements actually translate to in the real world would actually make. 

It’s a statement one makes when one is doing word manipulations. Where one sees a 35% on a piece of paper, doesn’t notice at all that this is from 11.9% to 8.7%, and decides that means that 35% of all patients will go from dying to not dying rather than 3.2%. 

No, seriously, they doubled down on this pretty hardcore. Check this out from deeper in that thread:

And now the FDA's Twitter account is promoting Hahn's disinfo.

Full quote from Hahn: "Let me just put that in perspective… What that means is if… 100 people who are sick with covid-19, 35 would have been saved because of the administration of plasma." https://t.co/XL8he5YWQ6

— Susan Simpson (@TheViewFromLL2) August 23, 2020

He did eventually walk it back and apologize. But think about the mindset required to get things this wrong. To not stop and think ‘wait, I don’t think 35% of patients even die, something’s wrong’ and check again. Until verified, everything such people say has to be presumed to likely be complete nonsense or propaganda, with no relation to the physical world. 

That doesn’t mean the treatment itself is bad.

A Miasma of Convalescent Plasma

So does convalescent plasma work?

Wired reports that 97,000 People Got Convalescent Plasma. Who Knows if It Works? The article contains lots of good information. Clearly epic ball droppings occurred. No one wanted to fund the clinical trials. If you’re wondering, all we had was money, what could we possibly have done to help? That’s something you could have done to help. We had tests ready to go, with no ethical issues, all they needed was funding. They didn’t happen, so now ‘no one knows if it works.’

Of course, that’s complete nonsense. It works.

It’s not the major breakthrough the President claimed it was. The range of possible effectiveness is ‘somewhat’ to ‘actually quite a bit’ but it’s still expensive in several senses to administer and all but certain to be supplanted by superior treatments soon. As that second link explains, convalescent plasma is a classic emergency stopgap strategy that is used until better options can come online. 

Still, it works. If someone is a patient and the plasma is available, until better options come along, this is not medical advice because legal reasons, but there’s no reason I can see to not to take the plasma. As everyone says, it’s a historically safe treatment and seems clearly safe once more. In terms of how effective it is, we know that patients that took it earlier did better than those who took it later, and those who had a stronger dose did better than those who had a weaker dose. 

Which likely means that even the strongest doses were probably not as big as they should have been if we have access to unlimited plasma. More dakka!

People can protest that ‘there are no RCTs’ all they like. That doesn’t make Bayes law go away. This one is very cut and dry. 

The issue is that if one is doing a tradeoff between the logistical costs of treatment versus spending those resources elsewhere, it’s hard to know what value to put down, but a good estimate does not seem that hard. The 35% number seems large in the sense that it is somewhat cherry picked, but it also seems like there’s a lot of room for improvement in the protocol in terms of timing, quantity/quality and procedure, as we learn more. So if we can get our testing done in a timely manner and get the treatment to patients quickly, my guess is this is good for at least that much. Which is a big forking deal.

Of course, a better treatment for those who get hospitalized if anything makes it harder to squash the virus, rather than easier, given the resulting changes in behavior. So this doesn’t put us any closer to normal. Still, great news as far as it goes.

FDA Fails To Approve Tests, People Die and Economy Collapses

A ‘mid-mortem’ came out this week on the FDA’s test approval policy. It mostly reinforces what we already mostly knew – the FDA made an active choice to shut down testing and force all action to go through proper channels, refusing to use its discretion, and then continued to require constantly changing actions and paperwork in order to get approval, and then did not prioritize the handling of said actions and paperwork on their end. Nor did it communicate its changing priorities and rules to those who needed to follow them. 

Many companies report waiting for months for word from the FDA, after filling out what they believed were the necessary requirements and paperwork, only to have their tests stranded in limbo. 

The biggest news there is that this process wasn’t only wholly unnecessary and monumentally destructive, but according to an HHS statement on August 19, much of it was not even within FDA’s jurisdictional rights to impose those restrictions. Many of its actions were without a legal basis.   

Once again: FDA delenda est. 

Musician Plays For Crowd

In ‘someone actually did any study of anything at all, so I’ll take it’ news, Coronavirus: Germany puts on crowded concerts to study risks

To quote the article: The concert study, called Restart-19, was created “to investigate the conditions under which such events can be carried out despite the pandemic,” researchers said.

As far as I can tell, what they studied was where people stood during the concert?

It’s not clear what else they could have done with this setup. It’s considered unethical to let anyone assume Covid-19 risk in order to help anyone else, so they made sure to test everyone for Covid-19 before the concert. At which point, they studied… the complete non-spreading of Covid-19 that took place, since everyone had tested negative? That does not seem super useful.

If you want to ‘investigate the conditions under which such events can be carried out’ and you’ve already decided the answer is ‘by testing everyone before the concert’ then I believe you have your answer. The good news is all we have to do is let people use reasonable and cheap tests, and then all that testing can be funded with less of an extra fee than the Ticketmaster ‘online purchase’ surcharge on tickets. 

Area Couple Does Math, Also Each Other, Probably Didn’t Actually Do Math

A common question I get asked, and also that I ask myself, is what activities are how risky (Google thinks the ‘are how risky’ is a grammar error and wants to correct it to ‘are risky,’ which explains a lot of how we’re in this mess!), and whether a given activity is a sane or responsible thing to be doing given the pandemic.

What about dating and casual sex, or sex in general? (link is to Washington Post)  

Several times there have been fun news items about municipalities trying to ban three ways, or encourage glory holes, or other similar things. I never thought any of those attempts would accomplish anything, but I was always both amused and delighted by such efforts. They represent an attempt to think about the physical world and the consequences of actions, and make decisions on that basis. 

Setting aside that casual sex and dating and the pursuit of happiness in general is considered by the Very Serious People to be ‘inessential’ making it bad to ‘take risk,’ how risky is it, and are there steps that one can take to limit that risk?

The upper bound on risk is that you’re completely exposed to one person. If they have it, you get it. It’s hard to think of something that would more reliably spread Covid-19 than an extended french kiss. That certainly has a leg up on ‘droplet spread’ and is a much scarier prospect than the sexual act itself. 

It seems very reasonable to pick your spots and then say ‘you know what, I don’t want to worry about this and it’s not clear in practice I can reduce that risk much further with precautions that would be useful, so we’ll just accept that risk and do whatever.’ Or perhaps you can make an effort to avoid anything touching anyone’s nose or mouth, or even leave the masks fully on while everything else comes off. To each their own.

But is it reasonable to expose yourself completely to one person in order to get lucky? If everyone is young and has been mostly isolating, and has no symptoms, this seems like it ends up pretty far down on the actual list of downsides to what you’re about to do. If the encounter would have been super exciting before, and again if everyone is otherwise being responsible and isn’t high risk, judicious amounts of this seem to me like an efficient use of one’s (and society’s) risk points. But keep in mind that doing it more with more different people dramatically increases the risk per encounter, so let’s not go too crazy. 

Let’s Go, Mets

My team, The New York Mets, missed five games because one player and one staff member tested positive for Covid-19. Of course, this happened in Miami. Out of an abundance of caution, the team returned to New York and quarantined for five days before resuming play.

It’s good to see baseball acting responsibly. If anything, this was more caution than necessary. 

Since more sports games are now being cancelled for purely symbolic protest reasons than for Covid-19 outbreaks, I’d say the restart of sports has been a smashing success.

The Simulacra of a Mask

While watching the disaster that is the Mets, I saw an ad imploring us all to wear face masks. It showed pictures of people wearing masks that said things like “for my husband” on them. I worry about the message ‘you wear a mask for other people’ given how people are sometimes, but mostly I don’t worry about this. Then the slogan across the screen: Show You Care. Wear a Mask.

So we’ve fully moved on to simulacra level 3

We could go with the Level 1 message that masks prevent infection. We could go with the level 2 message that they don’t, as many have tried at various points for various reasons, or that they are way more effective than they are, which has also been tried. Such messages are false, but they rest on the premise that people will decide whether masks help, and then do what is helpful and not what is unhelpful. 

Instead, we are being given the message that we must signal that we care about others, therefore we must wear a mask. 

The mask is being advertised as being as similar to not playing a basketball game to show you care about police violence. The basketball game does not cause police violence. 

The message is, hey, we know that wearing masks makes today a little bit worse, but it’s worth it, because you need to virtue signal. Bad things are happening, so you need to give up your nice thing, take away nice things and pressure others to give up their nice things, or people will think you do not care. And maybe, just maybe, if enough of us show we care, then someone will care enough to actually forking do something useful. 

This is very important, so I want to say this as clearly as I can: Fuck. That. Shit.

Wearing a mask is vital to preventing Covid-19 infection. Wear a forking mask. 

But only because it works. 

If you don’t think it works, don’t do it. 

You Had One Job

CDC decides third time’s the charm, decides to try harder once more with feeling, insists it needs to join the Delenda Est club.

It’s making a strong case!

You are literally called the Centers for Disease Control. Your one job is to control disease. 

So I want to know what the flying fork is going on over there, given the new guidance they issued this week. First link is the guidance, second and third are news coverage.

An attempt at a “good faith” interpretation of the new testing guidelines, that you ‘do not necessarily need a test’ even if you have been within 6 feet of a known positive for fifteen minutes, is that the CDC is lying. That they are doing the exact same thing with testing that was previously done with masks. 

The plan would be to say: 

  1. Testing doesn’t work for you! 
  2. Save the testing for those who really need it!

Because there aren’t enough tests, or doing too many tests is slowing down our turnaround times.

An attempt at an actual good faith interpretation is that these tests aren’t actually useful. That if you get exposed that getting a test does not have value until you have symptoms. The argument goes that getting a negative test doesn’t mean you won’t become positive later, so you have to quarantine anyway, so what’s the point in bothering. Because that’s how humans work, and that’s how human lives function. And no one else might want to know if they had been exposed in turn.

An attempt at a not-so-good faith interpretation is that this is the result of pressure from the White House, who want to suppress testing so the numbers look superficially better. 

Those were the three I could come up with.

And we have our answer, which I saw after writing the rest of this section. The directive came under pressure from the White House.

Then there’s the other half of the new guidance, which states that travellers no longer need to quarantine on arrival. As one would expect, local officials everywhere seem to disagree, such as in California. I would be floored if New York lifted its travel restrictions.

Basically, the CDC is taking the two things we actually know how to do to stop the spread beyond modifying day to day normal behavior. Our four tactics are, essentially: Wear a mask, socially distance, test and contract trace, and restrict travel to low-infection areas. The CDC is saying that contact tracing and a lot of testing isn’t worth it so don’t bother, and restricting travel isn’t worth it. Except that those are the only tactics that have been shown, anywhere, to actually work. 

We need more testing. Even the money congress allocated for testing, paltry as it was, mostly hasn’t been spent. Test counts are declining rather than increasing over the last few weeks, which is alarming. And while we need more testing, it would make sense to tell people who are low priority that they can’t be tested. But this is something else, and the stand against quarantines on travel makes no sense except maybe as some effort by spiteful people to get back at the blue states, either by tricking them into following it, or by getting to argue they are not ‘following CDC guidelines.’ 

The problem with putting the CDC in the Delenda Est club is that the CDC actually does have its one job to do. We should be doing disease control. The problem with the FDA is that it is doing its job. The problem with the CDC is that it is NOT doing its job. Big difference. So we need to fix the problems, but alas, I can’t let the CDC into the club. Maybe we need to burn it down and start over, but we’d then need to rebuild it, faster, stronger, better. Under no circumstances should we salt the earth.

But for now, we should treat the CDC as having been captured by the White House, and the White House as fully committed to misinformation. Thus, the CDC is now fully committed to misinformation. Going forward, at least until Trump leaves office, treat anything and everything from the CDC with extreme caution and as plausibly motivated as propaganda efforts to re-elect the President.



Discuss

Technical model refinement formalism

Новости LessWrong.com - 27 августа, 2020 - 14:54
Published on August 27, 2020 11:54 AM GMT

Introduction

.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} This post aims to formalise the models and model changes/model splintering described in this post. As explained there, the idea is to have a meta-model sufficiently general to be able to directly capture the process of moving from one imperfect model to another.

A note on infinity

For simplicity of exposition, I'll not talk about issues of infinite sets, continuity, convergence, etc... Just assume that any infinite set that comes up is just a finite set, large enough for whatever practical purpose we need it for.

Features, worlds, environments

A model M is defined by three object, the set F of features, the set E of environments, and a probability distribution Q. We'll define the first two in this section.

Features

Features are things that might be true or not about worlds, or might take certain values in worlds. For example, "the universe is open" is a possible feature about our universe, "the temperature is 250K" is another possible feature, but instead of returning true or false, it returns the temperature value. Adding more details, such as "the temperature is 250K in room 3, at 12:01" show that features should also be able to take inputs: features are functions.

But what about "the frequency of white light"? That's something that makes sense in many models - white light is used extensively in many contexts, and light has a frequency. The problem with that statement is that light has multiple frequencies; so we should allow features to be, at least in some cases, multivalued functions.

To top that off, sometimes there will be no correct value for a function; "the height of white light" is something that doesn't mean anything. So features have to include partial functions as well.

Fortunately, multivalued and partial functions are even simpler than functions at the formal level: they are just relations. And since the sets in the relations can consist of a single element, in even more generality, a feature is a predicate on a set. We just need to know which set.

So, formally, a feature F∈F consists of an (implicit) label defining what F is (eg "open universe", "temperature in some location") and a set on which it is a predicate. Thus, for example, the features above could be:

  1. Fopen universe={0} (features which are simply true or false are functions of a single element).
  2. Ftemperature={R+}.
  3. Ftemperature at location and time={L×T×R+}, for some set L of locations and T of possible times.
  4. Ffrequency of specific light={R+}.
  5. Fheight of object={O×R+} for O a set of objects.

Note these definitions are purely syntactic, not semantic: they don't have any meaning. Indeed, as sets, Ftemperature and Ffrequency of specific light are identical. Note also that there are multiple ways of defining the same things; instead of a single feature Ftemperature at location and time, we could have a whole collection of Ftemperature at land t for all (l,t)∈L×T.

Worlds

In Abram's orthodox case against utility functions he talks about the Jeffrey-Bolker axioms, which allows the construction of preferences from events without needing full worlds at all.

Similarly, this formalism is not focused on worlds, but it can be useful to define the full set of worlds for a model. This is simply the possible values that all features could conceivably take; so, if ¯¯¯¯¯F=⊔FF is the disjoint union of all features in F (seen as sets), the set of worlds W is just W=2¯¯¯¯F, the powerset of ¯¯¯¯¯F - equivalently, the set of all functions from ¯¯¯¯¯F to {True,False}.

So W just consists of all things that could be conceivably distinguished by the features. If we need more discrimination than this - just add more features.

Environments

The set of environments is a subset E of W, the set of worlds (though it need not be defined via W; it's a set of functions from ¯¯¯¯¯F to {True,False}.

Though this definition is still syntactic, it starts putting some restrictions on what the semantics could possibly be, in the spirit of this post.

For example, E could restrict to situations where Ftemperature is a single valued function, while Ffrequency of specific light is allowed to be multivalued. And similarly, Fheight of a specific oject takes no defined values on anything in the domain of Ffrequency of specific light.

Probability

The simplest way of defining Q is as a probability distribution over E.

This means that, if E1 and E2 are subsets of E, we can define the conditional probability

Q(E1∣E2)=Q(E1∩E2∣E2).

Once we have such a probability distribution, then, if the set of features is rich enough, this puts a lot more restrictions on the meaning that these features could have, going a lot of the way towards semantics. For example, if Q captures the ideal gas laws, then there is a specific relation between temperature, pressure, volume, and amount of substance - whatever those features are labelled.

In general, we'd want Q to be expressible in a simple way from the set F of features; that's the point of having those features in the first place.

Broader definition of "probability"

The plan for this meta-formalism is to allow transition from imperfect models to other imperfect models. So requiring that they have a probability distribution over all of E may be too much to ask.

In practice, all that is needed is expressions of the type Q(E1∣E2). And these may not be needed for all E1, E2. For example, to go back to the ideal gas laws, it makes perfect sense that we can deduce temperature from the other three features. But what if E2 just fixed the volume - can we deduce the pressure from that?

With Q as a prior over E, we can, by getting the pressure and amount of substance from the prior. But many models don't include these priors, and there's no reason to avoid those.

So, in the more general case, instead of E⊂W, define E⊂2W×2W, so that, for all (E1,E2)∈E, the following probability is defined:

Q(E1∣E2).

To insure consistency, we can require Q to follow axioms similar to the two-valued probabilities appendix *IV in Popper's "Logic of Scientific Discovery".

In full generality, we might need an even more general or imperfect definition of Q. But I'll leave this aside for the moment, and assume the simpler case where Q is a distribution over E.

Refinement

Here we'll look at how one can improve a model. Obviously, one can get a better Q, or a more expansive E, or a combination of these. Now, we haven't talked much about the quality of Q, and we'll leave this underdefined. Say that Q∗⪰Q means that Q∗ is 'at least as good as Q'. The 'at least as good' is specified by some mix of accuracy and simplicity.

More expansive E means that the environment of the improvement can be bigger. But in order for something to be "bigger", we need some identification between the two environments (which, so far, have just been defined as subsets of the powerset of feature values).

So, let M=(F,E,Q) and M∗=(F∗,E∗,Q∗) be models, let E∗0 be a subset of E∗, and let q be a surjective map from E∗0 to E (for an e∈E, think of q−1(e)⊂E∗0, the preimage of q, as the set of all environments in E∗ that correspond to e).

We can define Q∗0 on E in the following manner: if E1 and E2 are subsets of E, define

Q∗0(E1∣E2)=Q∗(q−1(E1)∣q−1(E2)).

Then q defines M∗ as a refinement of M if:

  • Q∗0⪰Q.
Refinement examples

Here are some examples of different types of refinements:

  1. Q-improvement: F=F∗, E=E∗, Q∗⪰Q (eg using the sine of the angle rather than the angle itself for refraction).
  2. Environment extension: F=F∗, E⊊E∗, E∗0=E with q the identity, Q∗=Q on E (eg moving from a training environment to a more extensive test environment).
  3. Natural extension: environment extension where Q is simply defined in terms of F on E, and this extends to Q∗ on E∗ (eg extending Newtonian mechanics from the Earth to the whole of the solar system).
  4. Non-independent feature extension: F⊊F∗. Let πF be the map that takes an element of W∗ and maps it to W by restricting[1] to features in F. Then πF=q on E∗0, and Q∗0=Q (eg adding electromagnetism to Newtonian mechanics).
  5. Independent feature extension: as a non-independent feature extension, but E∗0=E∗, and the stronger condition for Q∗ that Q(E1∣E2)=Q∗(q−1(E1)∣E∗2) for any E∗2 with q(E∗2)=E2 (eg non-colliding planets modelled without rotation, changing to modelling them with (mild) rotation).
  6. Feature refinement: F⊊F∗ (moving from the ideal gas models to the van der Waals equation).
  7. Feature splintering: when there is no single natural projection E∗→E that extends q (eg Blegg and Rube generalisation, happiness and human smile coming apart, inertial mass in general relativity projected to Newtonian mechanics...)
  8. Reward function splintering: no single natural extension of the reward function on E from E′=q−1(E) to all of E∗ (any situation where a reward function, seen as a feature, splinters).
Reward function: refactoring and splintering Reward function refactoring

Let M∗={F∗,E∗,Q∗} be a refinement of M={F,E,Q} (via q), and let R be a reward function defined on E.

A refactoring of R on M∗, is a reward function R∗ on E∗ such that for all e∗∈E∗, R(q(e∗))=R∗(e∗)). A natural refactoring is an extension of R is a refactoring that satisfies some naturalness or simplicity properties. For example, if R is the momentum of an object in M, and if momentum still makes sense in M∗, then this should be a natural refactoring.

Reward function splintering

If there does not exist a unique natural refactoring of R on M∗, then the refinement from M to M∗ splinters R.

Feature splintering

Let R be the indicator function for a feature being equal to some element or in some range. If R splinters in a refinement, then so does that feature.

  1. Note that W∗ is the set of all functions from ¯¯¯¯¯F∗ to {True,False}. Since F⊂F∗, ¯¯¯¯¯F=⊔FF⊂⊔F∗F=¯¯¯¯¯F∗. Then we can project from W∗ to W by restricting a function to its values on ¯¯¯¯¯F. ↩︎



Discuss

Model splintering: moving from one imperfect model to another

Новости LessWrong.com - 27 августа, 2020 - 14:53
Published on August 27, 2020 11:53 AM GMT

1. The big problem

In the last few months, I've become convinced that there is a key meta-issue in AI safety; a problem that seems to come up in all sorts of areas..mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}

It's hard to summarise, but my best phrasing would be:

  • Many problems in AI safety seem to be variations of "this approach seems safe in this imperfect model, but when we generalise the model more, it becomes dangerously underdefined". Call this model splintering.
  • It is intrinsically worth studying how to (safely) transition from one imperfect model to another. This is worth doing, independently of whatever "perfect" or "ideal" model might be in the background of the imperfect models.

This sprawling post will be presenting examples of model splintering, arguments for its importance, a formal setting allowing us to talk about it, and some uses we can put this setting to.

1.1 In the language of traditional ML

In the language of traditional ML, we could connect all these issues to "out-of-distribution" behaviour. This is the problems that algorithms encounter when the set they are operating on is drawn from a different distribution than the training set they were trained on.

Humans can often see that the algorithm is out-of-distribution and correct it, because we have a more general distribution in mind than the one the algorithm was trained on.

In these terms, the issues of this post can be phrased as:

  1. When the AI finds itself mildly out-of-distribution, how best can it extend its prior knowledge to the new situation?
  2. What should the AI do if it finds itself strongly out-of-distribution?
  3. What should the AI do if it finds itself strongly out-of-distribution, and humans don't know the correct distribution either?
1.2 Model splintering examples

Let's build a more general framework. Say that you start with some brilliant idea for AI safety/alignment/effectiveness. This idea is phrased in some (imperfect) model. Then "model splintering" happens when you or the AI move to a new (also imperfect) model, such that the brilliant idea is undermined or underdefined.

Here are a few examples:

  • You design an AI CEO as a money maximiser. Given typical assumptions about the human world (legal systems, difficulties in one person achieving massive power, human fallibilities), this results in an AI that behaves like a human CEO. But when those assumptions fail, the AI can end up feeding the universe to a money-making process that produces nothing of any value.
  • Eliezer defined "rubes" as smooth red cubes containing palladium that don't glow in the dark. "Bleggs", on the other hand, are furred blue eggs containing vanadium that glow in the dark. To classify these, we only need a model with two features, "rubes" and "bleggs". Then along comes a furred red egg containing vanadium that doesn't glow in the dark. The previous model doesn't know what to do with it, and if you get a model with more features, it's unclear what to do with this new object.
  • Here are some moral principles from history: honour is important for anyone. Women should be protected. Increasing happiness is important. These moral principles made sense in the world in which they were articulated, where features like "honour", "gender", and "happiness" are relatively clear and unambiguous. But the world changed, and the models splintered. "Honour" became hopelessly confused centuries ago. Gender is currently finishing its long splintering (long before we got to today, gender started becoming less useful for classifying people, hence the consequences of gender splintered a long time before gender itself did). Happiness, or at least hedonic happiness, is still well defined, but we can clearly see how this is going to splinter when we talk about worlds of uploads or brain modification.
  • Many transitions in the laws of physics - from the ideal gas laws to the more advanced van der Waals equations, or from Newtonain physics to general relativity to quantum gravity - will cause splintering if preferences were articulated in concepts that don't carry over well.
1.3 Avoiding perfect models

In all those cases, there are ways of improving the transition, without needing to go via some idealised, perfect model. We want to define the AI CEO's task in more generality, but we don't need to define this across every possible universe - that is not needed to restrain its behaviour. We need to distinguish any blegg from any rube we are likely to encounter, we don't need to define the platonic essence of "bleggness". For future splinterings - when hedonic happiness splinters, when we get a model of quantum gravity, etc... - we want to know what to do then and there, even if there are future splinterings subsequent to those.

And I think think that model splintering is best addressed directly, rather than using methods that go via some idealised perfect model. Most approaches seem to go for approximating an ideal: from AIXI's set of all programs, the universal prior, KWIK ("Knowing what it knows") learning with a full hypothesis class, Active Inverse Reward Design with its full space of "true" reward functions, to Q-learning which assumes any Markov decisions process is possible. Then the practical approaches rely on approximating this ideal.

Schematically, we can see M∞ as the ideal, Mi∞ as M∞ updated with information to time i, and Mi as an approximation of Mi∞. Then we tend to focus on how well Mi approximates Mi∞, and on how Mi∞ changes to Mi+1∞ - rather than on how Mi relates to Mi+1; the red arrow here is underanalysed:

2 Why focus on the transition?

But why is focusing on the Mi→Mi+1 transition important?

2.1 Humans reason like this

A lot has been written about image recognition programs going "out-of-distribution" (encountering situations beyond its training environment) or succumbing to "adversarial examples" (examples from one category that have the features of another). Indeed, some people have shown how to use labelled adversarial examples to improve image recognition.

You know what this reminds me of? Human moral reasoning. At various points in our lives, we humans seem to have pretty solid moral intuitions about how the world should be. And then, we typically learn more, realise that things don't fit in the categories we were used to (go "out-of-distribution") and have to update. Some people push stories at us that exploit some of our emotions in new, more ambiguous circumstances ("adversarial examples"). And philosophers use similarly-designed thought experiments to open up and clarify our moral intuitions.

Basically, we start with strong moral intuitions on under-defined features, and when the features splinter, we have to figure out what to do with our previous moral intuitions. A lot of developing moral meta-intuitions, is about learning how to navigate these kinds of transitions; AIs need to be able to do so too.

2.2 There are no well-defined overarching moral principles

Moral realists and moral non-realists agree more than you'd think. In this situation, we can agree on one thing: there is no well-described system of morality that can be "simply" implement in AI.

To over-simplify, moral realists hope to discover this moral system, moral non-realists hope to construct one. But, currently, it doesn't exist in an implementable form, nor is there any implementable algorithm to discover/construct it. So the whole idea of approximating an ideal is wrong.

All humans seem to start from a partial list of moral rules of thumb, rules that they then have to extend to new situations. And most humans do seem to have some meta-rules for defining moral improvements, or extensions to new situations.

We don't know perfection, but we do know improvements and extensions. So methods that deal explicitly with that are useful. Those are things we can build on.

2.3 It helps distinguish areas where AIs fail, from areas where humans are uncertain

Sometimes the AI goes out-of-distribution, and humans can see the error (no, flipping the lego block doesn't count as putting it on top of the other). There are cases when humans themselves go out-of-distribution (see for example siren worlds).

It's useful to have methods available for both AIs and humans in these situations, and to distinguish them. "Genuine human preferences, not expressed in sufficient detail" is not the same as "human preferences fundamentally underdefined".

In the first case, it needs more human feedback; in the second case, it needs to figure out way of resolving the ambiguity, knowing that soliciting feedback is not enough.

2.4 We don't need to make the problems harder

Suppose that quantum mechanics is the true underlying physics of the universe, with some added bits to include gravity. If that's true, why would we need a moral theory valid in every possible universe? It would be useful to have that, but would be strictly harder than one valid in the actual universe.

Also, some problems might be entirely avoided. We don't need to figure out the morality of dealing with a willing slave race - if we never encounter or build one in the first place.

So a few degrees of "extend this moral model in a reasonable way" might be sufficient, without needing to solve the whole problem. Or, at least, without needing to solve the whole problem in advance - a successful nanny AI might be built on these kinds of extensions.

2.5 We don't know how deep the rabbit hole goes

In a sort of converse to the previous point, what if the laws of physics are radically different from what we thought - what if, for example, they allow some forms of time-travel, or have some narrative features, or, more simply, what if the agent moves to an embedded agency model? What if hypercomputation is possible?

It's easy to have an idealised version of "all reality" that doesn't allow for these possibilities, so the ideal can be too restrictive, rather than too general. But the model splintering methods might still work, since it deals with transitions, not ideals.

Note that, in retrospect, we can always put this in a Bayesian framework, once we have a rich enough set of environments and updates rules. But this is misleading: the key issue is the missing feature, and figuring out what to do with the missing feature is the real challenge. The fact that we could have done this in a Bayesian way if we already knew that feature, is not relevant here.

2.6 We often only need to solve partial problems

Assume the blegg and rube classifier is an industrial robot performing a task. If humans filter out any atypical bleggs and rubes before it sees them, then the robot has no need for a full theory of bleggness/rubeness.

But what it the human filtering is not perfect? Then the classifier still doesn't need a full theory of bleggness/rubeness; it needs methods for dealing with the ambiguities it actually encounters.

Some ideas for AI control - low impact, AI-as-service, Oracles, ... - may require dealing with some model splintering, some ambiguity, but not the whole amount.

2.7 It points out when to be conservative

Some methods, like quantilizers or the pessimism approach rely on the algorithm having a certain degree of conservatism. But, as I've argued, it's not clear to what extent these methods actually are conservative, nor is it easy to calibrate them in a useful way.

Model splintering situations provide excellent points at which to be conservative. Or, for algorithms that need human feedback, but not constantly, these are excellent points to ask for that feedback.

2.8 Difficulty in capturing splintering from the idealised perspective

Generally speaking, idealised methods can't capture model splintering at the point we would want it to. Imagine an ontological crisis, as we move from classical physics to quantum mechanics.

AIXI can go over the transition fine: it shifts from a Turing machine mimicking classical physics observations, to one mimicking quantum observations. But it doesn't notice anything special about the transition: changing the probability of various Turing machines is what it does with observations in general; there's nothing in its algorithm that shows that something unusual has occurred for this particular shift.

2.9 It may help amplification and distillation

This could be seen as a sub-point of some of the previous two sections, but it deserves to be flagged explicitly, since iterated amplification and distillation is one of the major potential routes to AI safety.

To quote a line from that summary post:

  1. The proposed AI design is to use a safe but slow way of scaling up an AI’s capabilities, distill this into a faster but slightly weaker AI, which can be scaled up safely again, and to iterate the process until we have a fast and powerful AI.

At both "scaling up an AI's capabilities", and "distill this into", we can ask the question: has the problem the AI is working on changed? The distillation step is more of a classical AI safety issue, as we wonder whether the distillation has caused any value drift. But at the scaling up or amplification step, we can ask: since the AIs capabilities have changed, the set of possible environments it operates in has changed as well. Has this caused a splintering where the previously safe goals of the AI have become dangerous.

Detecting and dealing with such a splintering could both be useful tools to add to this method.

2.10 Examples of model splintering problems/approaches

At a meta level, most problems in AI safety seem to be variants of model splintering, including:

Almost every recent post I've read in AI safety, I've been able to connect back to this central idea. Now, we have to be cautious - cure-alls cure nothing, after all, so it's not necessarily a positive sign that everything seems to fit into this framework.

Still, I think it's worth diving into this, especially as I've come up with a framework that seems promising for actually solving this issue in many cases.

In a similar concept-space is Abram's orthodox case against utility functions, where he talks about the Jeffrey-Bolker axioms, which allows the construction of preferences from events without needing full worlds at all.

3 The virtues of formalisms

This post is dedicated to explicitly modelling the transition to ambiguity, and then showing what we can gain from this explicit meta-modelling. It will do with some formal language (made fully formal in this post), and a lot of examples.

Just as Scott argues that if it's worth doing, it's worth doing with made up statistics, I'd argue that if an idea is worth pursuing, it's worth pursuing with an attempted formalism.

Formalisms are great at illustrating the problems, clarifying ideas, and making us familiar with the intricacies of the overall concept. That's the reason that this post (and the accompanying technical post) will attempt to make the formalism reasonably rigorous. I've learnt a lot about this in the process of formalisation.

3.1 A model, in (almost) all generality

What do we mean by a model? Do we mean mathematical model theory? As we talking about causal models, or causal graphs? AIXI uses a distribution over possible Turing machines, whereas Markov Decision Processes (MDPs) sees states and actions updating stochastically, independently at each time-step. Unlike the previous two, Newtonian mechanics doesn't use time-steps but continuous times, while general relativity weaves time into the structure of space itself.

And what does it mean for a model to make "predictions"? AIXI and MDPs make prediction over future observations, and causal graphs are similar. We can also try running them in reverse, "predicting" past observations from current ones. Mathematical model theory talks about properties and the existence or non-existence of certain objects. Ideal gas laws make a "prediction" of certain properties (eg temperature) given certain others (eg volume, pressure, amount of substance). General relativity establishes that the structure of space-time must obey certain constraints.

It seems tricky to include all these models under the same meta-model formalism, but it would be good to do so. That's because of the risk of ontological crises: we want the AI to be able to continue functioning even if the initial model we gave it was incomplete or incorrect.

3.2 Meta-model: models, features, environments, probabilities

All of the models mentioned above share one common characteristic: once you know some facts, you can deduce some other facts (at least probabilistically). A prediction of the next time step, a retrodiction of the past, a deduction of some properties from other, or a constraint on the shape of the universe: all of these say that if we know some things, then this puts constraints on some other things.

So let's define F, informally, as the set of features of a model. This could be the gas pressure in a room, a set of past observations, the local curvature of space-time, the momentum of a particle, and so on.

So we can define a prediction as a probability distribution over a set of possible features F1, given a base set of features, F2:

Q(F1∣F2).

Do we need anything else? Yes, we need a set of possible environments for which the model is (somewhat) valid. Newtonian physics fails at extreme energies, speeds, or gravitational fields; we'd like to include this "domain of validity" in the model definition. This will be very useful for extending models, or transitioning from one model to another.

You might be tempted to define a set of "worlds" on which the model is valid. But we're trying to avoid that, as the "worlds" may not be very useful for understanding the model. Moreover, we don't have special access to the underlying reality; so we never know whether there actually is a Turing machine behind the world or not.

So define E, the environment on which the model is valid, as a set of possible features. So if we want to talk about Newtonian mechanics, F would be a set of Newtonian features (mass, velocity, distance, time, angular momentum, and so on) and E would be the set of these values where relativistic and quantum effects make little difference.

So see a model as

M={F,E,Q},

for F a set of features, E a set of environments, and Q a probability distribution. This is such that, for E1,E2⊂E, we have the conditional probability:

Q(E1∣E2).

Though Q is defined for E, we generally want it to be usable from small subsets of the features: so Q should be simple to define from F. And we'll often define the subsets Ei in similar ways; so E1 might be all environments with a certain angular momentum at time t=0, while E2 might be all environments with a certain angular momentum at a later time.

The full formal definition of these can be found here. The idea is to have a meta-model of modelling that is sufficiently general to apply to almost all models, but not one that relies on some ideal or perfect formalism.

3.3 Bayesian models within this meta-model

It's very easy to include Bayesian models within this formalism. If we have a Bayesian model that includes a set W of worlds with prior P, then we merely have to define a set of features F that is sufficient to distinguish all worlds in W: each world is uniquely defined by its feature values[1]. Then we can define E as W, and P on W becomes Q on E; the definitions of terms like Q(E1∣E2) is just P(E1∩E2)P(E1)/P(E2), per Bayes' rules (unless P(E2)=0, in which case we set that to 0).

4 Model refinement and splinterings

This section will look at what we can do with the previous meta-model, looking at refinement (how models can improve) and splintering (how improvements to the model can make some well-defined concepts less well-defined).

4.1 Model refinement

Informally, M∗={F∗,E∗,Q∗} is a refinement of model M={F,E,Q} if it's at least as expressive as M (it covers the same environments) and is better according to some criteria (simpler, or more accurate in practice, or some other measurement).

At the technical level, we have a map q from a subset E∗0 of E∗, that is surjective onto E. This covers the "at least as expressive" part: every environment in E exists as (possibly multiple) environments in E∗.

Then note that using q−1 as a map from subsets of E to subsets of E∗0, we can define Q∗0 on E via:

Q∗0(E1∣E2)=Q∗(q−1(E1)∣q−1(E2)).

Then this is a model refinement if Q∗0 is 'at least as good as' Q on E, according to our criteria[2].

4.2 Example of model refinement: gas laws

This post presents some subclasses of model refinement, including Q-improvements (same features, same environments, just a better Q), or adding new features to a basic model, called "non-independent feature extension" (eg adding classical electromagnetism to Newtonian mechanics).

Here's a specific gas law illustration. Let M={F,E,Q} be a model of an ideal gas, in some set of rooms and tubes. The F consists of pressure, volume, temperature, and amount of substance, and Q is the ideal gas laws. The E is the standard conditions for temperature and pressure, where the ideal gas law applies. There are multiple different types of gases in the world, but they all roughly obey the same laws.

Then compare with model M∗={F∗,E∗,Q∗}. The F∗ has all the features of F, but also includes the volume that is occupied by one mole of the molecules of the given substance. This allows Q∗ to express the more complicated van der Waals equations, which are different for different types of gases. The E∗ can now track situations where there are gases with different molar volumes, which include situations where the van der Waals equations differ significantly from the ideal gas laws.

In this case E∗0⊂E∗, since we now distinguish environments that we previously considered identical (environments with same features except for having molar volumes). The q is just projecting down by forgetting the molar volume. Then since Q∗0=Q∗ (van der Waals equations averaged over the distribution of molar volumes) is at least as accurate as Q (ideal gas law), this is a refinement.

4.3 Example of model refinement: rubes and bleegs

Let's reuse Eliezer's example of rubes ("red cubes") and bleggs ("blue eggs").

Bleggs are blue eggs that glow in the dark, have a furred surface, and are filled with vanadium. Rubes, in contrast, are red cubes that don't glow in the dark, have a smooth surface, and are filled with palladium:

Define M by having F={red,smooth}, E is the set of all bleggs and rubes in some situation, and Q is relatively trivial: it predicts that an object is red/blue if and only if is smooth/furred.

Define M1 as a refinement of M, by expanding F to F1={red,smooth,cube,dark}. The projection q:E∗→E is given by forgetting about those two last features. The Q1 is more detailed, as it now connects red-smooth-cube-dark together, and similarly for blue-furred-egg-glows.

Note that E1 is larger than E, because it includes, e.g., environments where the cube objects are blue. However, all these extra environments have probability zero.

4.4 Reward function refactoring

Let R be a reward function on M (by which we mean that R is define on F, the set of features in M), and M∗ a refinement of M.

A refactoring of R for M∗ is a reward function R∗ on the features F∗ such that for any e∗∈E∗0, R∗(e∗)=R(q(e∗)).

For example, let M and M1 be from the rube/blegg models in the previous section. Let Rred on M simply count the number of rubes - or, more precisely, counts the number of objects to which the feature "red" applies.

Let R1red be the reward function that counts the number of objects in M1 to which "red" applies. It's clearly a refactoring of Rred.

But so is R1smooth, the reward function that counts the number of objects in M1 to which "smooth" applies. In fact, the following is a refactoring of Rred, for all α+β+γ+δ=1:

αR1red+βR1smooth+γR1cube+δR1dark.

There are also some non-linear combinations of these features that refactor R, and many other variants (like the strange combinations that generate concepts like grue and bleen).

4.5 Reward function splintering

Model splintering, in the informal sense, is what happens when we pass to a new models in a way that the old features (or a reward function defined by the old features) no longer apply. It is similar to the web of connotations breaking down, an agent going out of distribution, or the definitions of Rube and Blegg falling apart.

  • Preliminary definition: If M∗ is a refinement of M and R a reward function on M, then M∗ splinters R if there are multiple refactorings of R on M∗ that disagree on elements of E∗ of non-zero probability.

So, note that in the rube/blegg example, M1 is not a splintering of Rred: all the refactorings are the same on all bleggs and rubes - hence on all elements of E1 of non-zero probability.

We can even generalise this a bit. Let's assume that "red" and "blue" are not totally uniform; there exists some rubes that are "redish-purple", while some bleggs are "blueish-purple". Then let M2 be like M1, except the colour feature can have four values: "red", "redish-purple", "blueish-purple", and "blue".

Then, as long as rubes (defined, in this instance, by being smooth-dark-cubes) are either "red" or "redish-purple", and the bleggs are "blue", or "blueish-purple", then all refactorings of Rred to M2 agree - because, on the test environment, Rred on F perfectly matches up with R2red+R2redish-purple on F2.

So adding more features does not always cause splintering.

4.6 Reward function splintering: "natural" refactorings

The preliminary definition runs into trouble when we add more objects to the environments. Define M3 as being the same as M2, except that E3 contains one extra object, o+; apart from that, the environments typically have a billion rubes and a trillion bleggs.

Suppose o+ is a "furred-rube", i.e. a red-furred-dark-cube. Then R3red and R3smooth are two different refactorings of Rred, that obviously disagree on any environment that contains o+. Even if the probability of o+ is tiny (but non-zero), then M3 splinters R.

But things are worse than that. Suppose that o+ is fully a rube: red-smooth-cube-dark, and even contains palladium. Define (R3red)′ as being counting the number of red objects, except for o+ specifically (again, this is similar to the grue and bleen arguments against induction).

Then both (R3red)′ and R3red are refactorings of Rred, so M3 still splinters Rred, even when we add another exact copy of the elements in the training set. Or even if we keep the training set for a few extra seconds, or add any change to the world.

So, for any M∗ a refinement of M, and R a reward function on E, let's define "natural refactorings" of R:

  • The reward function R∗ is a natural refactoring of R if it's a reward function on M∗ with:
  1. R∗≈R∘q on E∗0, and
  2. R∗ can be defined simply from F∗ and R,
  3. the F∗ themselves are simply defined.

This leads to a full definition of splintering:

  • Full definition: If M∗ is a refinement of M and R a reward function on M, then M∗ splinters R if 1) there are no natural refactoring of R on M∗, or 2) there are multiple natural refactorings R∗ and R∗′ of R on M∗, such that R∗≉R∗′.

Notice the whole host of caveats and weaselly terms here; R∗≈R∘q, "simply" (used twice), and R∗≉R∗′. Simply might mean algorithmic simplicity, but ≈ and ≉ are measures of how much "error" we are willing to accept in these refactorings. Given that, we probably want to replace ≈ and ≉ with some measure of non-equality, so we can talk about the "degree of naturalness" or the "degree of splintering" of some refinement and reward function.

Note also that:

  • Different choices of refinements can result in different natural refactorings.

An easy example: it makes a big difference whether a new feature is "temperature", or "divergence from standard temperatures".

4.7 Splintering training rewards

The concept of "reward refactoring" is transitive, but the concept of "natural reward refactoring" need not be.

For example, let Et be a training environment where red/blue ⟺ cube/egg, and Eg be a general environment where red/blue is independent of cube/egg. Let F1 be a feature set with only red/blue, and F2 a feature set with red/blue and cube/egg.

Then define M1t as using F1 in the training environment, M2g as using F2 in the general environment; M1g and M2t are defined similarly.

For these models, M1g and M2t are both refinements of M1t, while M2g is a refinement of all three other models. Define R1t as the "count red objects" reward on M1t. This has a natural refactoring to R1g on M1g, which counts red objects in the general environment.

And R1g has a natural refactoring to R2g on M2g, which still just counts the red objects in the general environment.

But there is no natural refactoring from R1t directly to M2g. That's because, from F2's perspective, R1t on M1t might be counting red objects, or might be counting cubes. This is not true for R1g on M1g, which is clearly only counting red objects.

Thus when a reward function come from a training environment, we'd want our AI to look for splinterings directly from a model of the training environment, rather than from previous natural refactorings.

4.8 Splintering features and models

We can also talk about splintering features and models themselves. For M={F,E,Q}, the easiest way is to define a reward function RF,sF as being the indicator function for feature F∈F being in the set SF.

Then a refinement M∗ splinters the feature F if it splinters some RF,SF.

The refinement M∗ splinters the model M if it splinters at least one of its features.

For example, if M is Newtonian mechanics, including "total rest mass" and M∗ is special relativity, then M∗ will splinter "total rest mass". Other examples of feature splintering will be presented in the rest of this post.

4.9 Preserved background features

A reward function developed in some training environment will ignore any feature that is always present or always absent in that environment. This allows very weird situations to come up, such as training an AI to distinguish happy humans from sad humans, and it ending up replacing humans with humanoid robots (after all, both happy and sad humans were equally non-robotic, so there's no reason not to do this).

Let's try and do better than that. Assume we have a model M={F,E,Q}, with a reward function Rτ defined on E (Rτ and E can be seen as the training data).

Then the feature-preserving reward function RM, is a function that constrains the environments to have similar feature distributions as E and Q. There are many ways this could be defined; here's one.

For an element e∈E, just define

RM(e)=log(Q(e)).

Obviously, this can be improved; we might want to coarse-grain F, grouping together similar worlds, and possibly bounding this below to avoid singularities.

Then we can use this to get the feature-preserving version of Rτ, which we can define as

RMτ=(maxRτ−Rτ)⋅RM,

for maxRτ the maximal value of Rτ on E. Other options can work as well, such as Rτ+αRMτ for some constant 0">α>0.

Then we can ask an AI to use RMτ as its reward function, refactoring that, rather than Rτ.

  • A way of looking at it: a natural refactoring of a reward function Rτ will preserve all the implicit features that correlate with Rτ. But RMτ will also preserve all the implicit features that stay constant when Rτ was defined. So if Rτ measures human happiness vs human unhappiness, a natural refactoring of it will preserves things like "having higher dopamine in their brain". But a natural refactoring of RMτ will also preserve things like "having a brain".
4.10 Partially preserved background features

The RMτ is almost certainly too restrictive to be of use. For example, if time is a feature, then this will fall apart when the AI has to do something after the training period. If all the humans in a training set share certain features, humans without those features will be penalised.

There are at least two things we can do to improve this. The first is to include more positive and negative examples in the training set; for example, if we include humans and robots in our training set - as positive and negative examples, respectively - then this difference will show up in Rτ directly, so we won't need to use RMτ too much.

Another approach would be to explicitly allow certain features to range beyond their typical values in M, or allow highly correlated variables explicitly to decorrelate.

For example, though training during a time period t to t′, we could explicitly allow time to range beyond these values, without penalty. Similarly, if a medical AI was trained on examples of typical healthy humans, we could decorrelate functioning digestion from brain activity, and get the AI to focus on the second[3].

This has to be done with some care, as adding more degrees of freedom adds more ways for errors to happen. I'm aiming to look further at this issue in later posts.

5 The fundamental questions of model refinements and splintering

We can now rephrase the out-of-distribution issues of section 1.1 in terms of the new formalism:

  1. When the AI refines its model, what would count as a natural refactoring of its reward function?
  2. If the refinements splinter its reward function, what should the AI do?
  3. If the refinements splinter its reward function, and also splinters the human's reward function, what should the AI do?
6 Examples and applications

The rest of this post is applying this basic framework, and its basic insights, to various common AI safety problems and analyses. This section is not particularly structured, and will range widely (and wildly) across a variety of issues.

6.1 Extending beyond the training distribution

Let's go back to the blegg and rube examples. A human supervises an AI in a training environment, labelling all the rubes and bleggs for it.

The human is using a very simple model, MH={FH,Et,Q}, with the only feature being the colour of the object, and Et being the training environment.

Meanwhile the AI, having more observational abilities and no filter as to what can be ignored, notices their colour, their shape, their luminance, and their texture. It doesn't know MH, but is using model M1AI={F1,E1t,Q1}, where F1AI covers those four features (note that M1AI is a refinement of MH, but that isn't relevant here).

Suppose that the AI is trained to be rube-classifier (and hence a blegg classifier by default). Let RF be the reward function that counts the number of objects, with feature F, that the AI has classified as rubes. Then the AI could learn many different reward function in the training environment; here's one:

R1=R1cube+0.5R1smooth+0.5R1dark−R1red.

Note that, even though this gets the colour reward completely wrong, this reward matches up with the human's assessment on the training environment.

Now the AI moves to the larger testing environment E2, and refines its model minimally to M2AI={F1,E2,Q1} (extending R1 to R2 in the obvious way).

In E2, the AI sometimes encounters objects that it can only see through their colour. Will this be a problem, since the colour component of R2 is pointing in the wrong direction?

No. It still has Q1, and can deduce that a red object must be cube-smooth-dark, so R2 will continue treating this as a rube[4].

6.2 Detecting going out-of-distribution

Now imagine the AI learns about the content of the rubes and bleggs, and so refines to a new model that includes vanadium/palladium as a feature in M3AI.

Furthermore, in the training environment, all rubes have palladium and all bleggs have vanadium in them. So, for M3AI a refinement of M1AI, q−1(E1AI)⊂E3AI has only palladium-rubes and vanadium-bleggs. But in E3AI, the full environment, there are rather a lot of rubes with vanadium and bleggs with palladium.

So, similarly to section 4.7, there is no natural refactoring of the rube/blegg reward in M1AI, to M3AI. That's because F3AI, the feature set of M3AI, includes vanadium/palladium which co-vary with the other rube/blegg features on the training environment (q^{-1}(\E_{AI}^1)), but not on the full environment of E3AI.

So looking for reward splintering from the training environment is a way of detecting going out-of-distribution - even on features that were not initially detected in the training distribution, by either the human nor the AI.

6.3 Asking humans and Active IRL

Some of the most promising AI safety methods today rely on getting human feedback[5]. Since human feedback is expensive, as in it's slow and hard to get compared with almost all other aspects of algorithms, people want to get this feedback in the most efficient ways possible.

A good way of doing this would be to ask for feedback when the AI's current reward function splinters, and multiple options are possible.

A more rigorous analysis would look at the value of information, expected future splinterings, and so on. This is what they do in Active Inverse Reinforcement Learning; the main difference is that AIRL emphasises an unknown reward function with humans providing information, while this approach sees it more as an known reward function over uncertain features (or over features that may splinter in general environments).

6.4 A time for conservatism

I argued that many "conservative" AI optimising approaches, such as quantilizers and pessimistic AIs, don't have a good measure of when to become more conservative; their parameters q and β don't encode useful guidelines for the right degree of conservatism.

In this framework, the alternative is obvious: AIs should become conservative when their reward functions splinter (meaning that the reward function compatible with the previous environment has multiple natural refactorings), and very conservative when they splinter a lot.

This design is very similar to Inverse Reward Design. In that situation, the reward signal in the training environment is taken as information about the "true" reward function. Basically they take all reward functions that could have given the specific reward signals, and assume the "true" reward function is one of them. In that paper, they advocate extreme conservatism at that point, by optimising the minimum of all possible reward functions.

The idea here is almost the same, though with more emphasis on "having a true reward defined on uncertain features". Having multiple contradictory reward functions compatible with the information, in the general environment, is equivalent with having a lot of splintering of the training reward function.

6.5 Avoiding ambiguous distant situations

The post "By default, avoid ambiguous distant situations" can be rephrased as: let M be a model in which we have a clear reward function R, and let M2 be a refinement of this to general situations. We expect that this refinement splinters R. Let M1 be like M2, except with E1 smaller than E2, defined such that:

  1. An AI could be expected to be able to constrain the world to be in E1, with high probability,
  2. The M1 is not a splintering of R.

Then that post can be summarised as:

  • The AI should constrain the world to be in E1 and then maximise the natural refactoring of R in M1.
6.6 Extra variables

Stuart Russell writes:

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.

The approach in sections 4.9 and 4.10 explicitly deals with this.

6.7 Hidden (dis)agreement and interpretability

Now consider two agents doing a rube/blegg classifications task in the training environment; each agent only models two of the features:

Despite not having a single feature in common, both agents will agree on what bleggs and rubes are, in the training environment. And when refining to a fuller model that includes all four (or five) of the key features, both agents will agree as to whether a natural refactoring is possible or not.

This can be used to help define the limits of interpretability. The AI can use its own model, and its own designed features, to define the categories and rewards in the training environment. These need not be human-parsable, but we can attempt to interpret them in human terms. And then we can give this interpretation to the AI, as a list of positive and negative examples of our interpretation.

If we do this well, the AI's own features and our interpretation will match up in the training environment. But as we move to more general environments, these may diverge. Then the AI will flag a "failure of interpretation" when its refactoring diverges from a refactoring of our interpretation.

For example, if we think the AI detects pandas by looking for white hair on the body, and black hair on the arms, we can flag lots of examples of pandas and that hair pattern (and non-pandas and unusual hair patterns. We don't use these examples for training the AI, just to confirm that, in the training environment, there is a match between "AI-thinks-they-are-pandas" and "white-hair-on-arms-black-hair-on-bodies".

But, in an adversarial example, the AI could detect that, while it is detecting gibbons, this no longer matches up with our interpretaion. A splintering of interpretations, if you want.

6.8 Wireheading

The approach can also be used to detect wireheading. Imagine that the AI has various detectors that allow it to label what the features of the bleggs and rubes are. It models the world with ten features: 5 features representing the "real world" versions of the features, and 5 representing the "this signal comes from my detector" versions.

This gives a total of 10 features, the 5 features "in the real world" and the 5 "AI-labelled" versions of these:

In the training environment, there was full overlap between these 10 features, so the AI might learn the incorrect "maximise my labels/detector signal" reward.

However, when it refines its model to all 10 features and environments where labels and underlying reality diverge, it will realise that this splinters the reward, and thus detect a possible wireheading. It could then ask for more information, or have an automated "don't wirehead" approach.

6.9 Hypotheticals, and training in virtual environments

To get around the slowness of the real world, some approaches train AIs in virtual environments. The problem is to pass that learning from the virtual environment to the real one.

Some have suggested making the virtual environment sufficiently detailed that the AI can't tell the difference between it and the real world. But, a) this involves fooling the AI, an approach I'm always wary of, and b) it's unnecessary.

Within the meta-formalism of this post, we could train the AI in a virtual environment which it models by M, and let it construct a model M′ of the real-world. We would then motivate the AI to find the "closest match" between M and M′, in terms of features and how they connect and vary. This is similar to how we can train pilots in flight simulators; the pilots are never under any illusion as to whether this is the real world or not, and even crude simulators can allow them to build certain skills[6].

This can also be used to allow the AI to deduce information from hypotheticals and thought experiments. If we show the AI an episode of a TV series showing people behaving morally (or immorally), then the episode need not be believable or plausible, if we can roughly point to the features in the episode that we want to emphasise, and roughly how these relate to real-world features.

6.10 Defining how to deal with multiple plausible refactorings

The approach for synthesising human preferences, defined here, can be rephrased as:

  • "Given that we expect multiple natural refactorings of human preferences, and given that we expect some of them to go disastrously wrong, here is one way of resolving the splintering that we expect to be better than most."

This is just one way of doing this, but it does show that "automating what AIs do with multiple refactorings" might not be impossible. The following subsection has some ideas with how to deal with that.

6.11 Global, large scale preferences

In an old post, I talked about the concept of "emergency learning", which was basically, "lots of examples, and all the stuff we know and suspect about how AIs can go wrong, shove it all in, and hope for the best". The "shove it all in" was a bit more structured than that, defining large scale preferences (like "avoid siren worlds" and "don't over-optimise") as constraints to be added to the learning process.

It seems we can do better than that here. Using examples and hypotheticals, it seems we could construct ideas like "avoid slavery", "avoid siren worlds", or "don't over-optimise" as rewards or positive/negative examples certain simple training environments, so that the AI "gets an idea of what we want".

We can then label these ideas as "global preferences". The idea is that they start as loose requirements (we have much more granular human-scale preferences than just "avoid slavery", for example), but, the more the world diverges from the training environment, the stricter they are to be interpreted, with the AI required to respect some softmin of all natural refactorings of these features.

In a sense, we'd be saying "prevent slavery; these are the features of slavery, and in weird worlds, be especially wary of these features".

6.12 Avoiding side-effects

Krakovna et. al. presented a paper on avoiding side-effects from AI. The idea is to have an AI maximising some reward function, while reducing side effects. So the AI would not smash vases or let them break, nor would it prevent humans from eating sushi.

In this environment, we want the AI to avoid knocking the sushi off the belt as it moves:

Here, in contrast, we'd want the AI to remove the vase from the belt before it smashes:

I pointed out some issues with the whole approach. Those issues were phrased in terms of sub-agents, but my real intuition is that syntactic methods are not sufficient to control side effects. In other words, the AI can't learn to do the right thing with sushis and vases, unless it has some idea of what these objects mean to us; we prefer sushis to be eaten and vases to not be smashed.

This can be learnt if the AI has a enough training examples, learning that eating sushi is a general feature of the environments it operates in, while vases being smashed is not. I'll return to this idea in a later post.

6.13 Cancer patients

The ideas of this post were present in implicit form in the idea of training an AI to cure cancer patients.

Using examples of successfully treated cancer patients, we noted they all shared some positive features (recuperating, living longer) and some incidental or negative features (complaining about pain, paying more taxes).

So, using the approach of section 4.9, we can designate that we want the AI to cure cancer; this will be interpreted as increasing all the features that correlate with that.

Using the explicit decorrelation of section 4.10, we can also explicitly remove the negative options from the desired feature sets, thus improving the outcomes even more.

6.14 The genie and the burning mother

In Eliezer's original post on the hidden complexity of wishes, he talks of the challenge of getting a genie to save your mother from a burning building:

So you hold up a photo of your mother's head and shoulders; match on the photo; use object contiguity to select your mother's whole body (not just her head and shoulders); and define the future function using your mother's distance from the building's center. [...]

You cry "Get my mother out of the building!", for luck, and press Enter. [...]

BOOM! With a thundering roar, the gas main under the building explodes. As the structure comes apart, in what seems like slow motion, you glimpse your mother's shattered body being hurled high into the air, traveling fast, rapidly increasing its distance from the former center of the building.

How could we avoid this? What you want is your mother out of the building. The feature "mother in building" must absolutely be set to false; this is a priority call, overriding almost everything else.

Here we'd want to load examples of your mother outside the building, so that the genie/AI learns the features "mother in house"/"mother out of house". Then it will note that "mother out of house" correlates with a whole lot of other features - like mother being alive, breathing, pain-free, often awake, and so on.

All those are good things. But there are some other features that don't correlate so well - such as the time being earlier, your mother not remembering a fire, not being covered in soot, not worried about her burning house, and so on.

As in the cancer patient example above, we'd want to preserve the features that correlate with the mother out of the house, while allowing decorrelation with the features we don't care about or don't want to preserve.

6.15 Splintering moral-relevant categories: honour, gender, and happiness

If the Antikythera mechanism had been combined with the Aeolipile to produce an ancient Greek AI, and Homer had programmed it (among other things) to "increase people's honour", how badly would things have gone?

If Babbage had completed the analytical engine as Victorian AI, and programmed it (among other things) to "protect women", how badly would things have gone?

If a modern programmer were to combine our neural nets into a superintelligence and program it (among other things) to "increase human happiness", how badly will things go?

There are three moral-relevant categories here, and it's illustrative to compare them: honour, gender, and hedonic happiness. The first has splintered, the second is splintering, and the third will likely splinter in the future.

I'm not providing solutions in this subsection, just looking at where the problems can appear, and encouraging people to think about how they would have advised Homer or Babbage to define their concepts. Don't think "stop using your concepts, use ours instead", because our concepts/features will splinter too. Think "what's the best way they could have extended their preferences even as the features splinter"?

  • 6.15.1 Honour

If we look at the concept of honour, we see a concept that has already splintered.

That article reads like a meandering mess. Honour is "face", "reputation", a "bond between an individual and a society", "reciprocity", a "code of conduct", "chastity" (or "virginity"), a "right to precedence", "nobility of soul, magnanimity, and a scorn of meanness", "virtuous conduct and personal integrity", "vengeance", "credibility", and so on.

What a basket of concepts! They only seem vaguely connected together; and even places with strong honour cultures differ in how they conceive of honour, from place to place and from epoch to epoch[7]. And yet, if you asked most people within those cultures about what honour was, they would have had a strong feeling it was a single, well defined thing, maybe even a concrete object.

  • 6.15.2 Gender

In his post the categories were made for man, not man for the categories, Scott writes:

Absolutely typical men have Y chromosomes, have male genitalia, appreciate manly things like sports and lumberjackery, are romantically attracted to women, personally identify as male, wear male clothing like blue jeans, sing baritone in the opera, et cetera.

But Scott is writing this in the 21st century, long after the gender definition has splintered quite a bit. In middle class middle class Victorian England[8], the gender divide was much stronger - in that, from one component of the divide, you could predict a lot more. For example, if you knew someone wore dresses in public, you knew that, almost certainly, they couldn't own property if they were married, nor could they vote, they would be expected to be in charge of the household, might be allowed to faint, and were expected to guard their virginity.

We talk nowadays about gender roles multiplying or being harder to define, but they've actually being splintering for a lot longer than that. Even though we could define two genders in 1960s Britain, at least roughly, that definition was a lot less informative than it was in Victorian-middle-class-Britain times: it had many fewer features strongly correlated with it.

  • 6.15.3 Happiness

On to happiness! Philosophers and others have been talking about happiness for centuries, often contrasting "true happiness", or flourishing, with hedonism, or drugged out stupor, or things of that nature. Often "true happiness" is a life of duty to what the philosopher wants to happen, but at least there is some analysis, some breakdown of the "happiness" feature into smaller component parts.

Why did the philosophers do this? I'd wager that it's because the concept of happiness was already somewhat splintered (as compared with a model where "happiness" is a single thing). Those philosophers had experience of joy, pleasure, the satisfaction of a job well done, connection with others, as well as superficial highs from temporary feelings. When they sat down to systematise "happiness", they could draw on the features of their own mental model. So even if people hadn't systematised happiness themselves, when they heard of what philosophers were doing, they probably didn't react as "What? Drunken hedonism and intellectual joy are not the same thing? How dare you say such a thing!"

But looking into the future, into a world that an AI might create, we can foresee many situations where the implicit assumptions of happiness come apart, and only some remain. I say "we can foresee", but it's actually very hard to know exactly how that's going to happen; if we knew it exactly, we could solve the issues now.

So, imagine a happy person. What do you think that they have in life, that are not trivial synonyms of happiness? I'd imagine they have friends, are healthy, think interesting thoughts, have some freedom of action, may work on worthwhile tasks, may be connected with their community, probably make people around them happy as well. Getting a bit less anthropomorphic, I'd also expect them to be a carbon-based life-form, to have a reasonable mix of hormones in their brain, to have a continuity of experience, to have a sense of identity, to have a personality, and so on.

Now, some of those features can clearly be separated from "happiness". Even ahead of time, I can confidently say that "being a carbon-based life-form" is not going to be a critical feature of "happiness". But many of the other ones are not so clear; for example, would someone without continuity of experience or a sense of identity be "happy"?

Of course, I can't answer that question. Because the question has no answer. We have our current model of happiness, which co-varies with all those features I listed and many others I haven't yet thought of. As we move into more and more bizarre worlds, that model will splinter. And whether we assign the different features to "happiness" or to some other concept, is a choice we'll make, not a well-defined solution to a well-defined problem.

However, even at this stage, some answers are clearly better than others; statues of happy people should not count, for example, nor should written stories describing very happy people.

6.16 Apprenticeship learning

In apprenticeship learning (or learning from demonstration), the AI would aim to copy what experts have done. Inverse reinforcement learning can be used for this purpose, by guessing the expert's reward function, based on their demonstrations. It looks for key features in expert trajectories and attempts to reproduce them.

So, if we had an automatic car driving people to the airport, and fed it some trajectories (maybe ranked by speed of delivery), it would notice that passengers would also arrive alive, with their bags, without being pursued by the police, and so on. This is akin to section 4.9, and would not accelerate blindly to get there as fast as possible.

But the algorithm has trouble getting to truly super-human performance[9]. It's far too conservative, and, if we loosen the conservatism, it doesn't know what's acceptable and what isn't, and how to trade these off: since all passengers survived and the car was always painted yellow, their luggage intact in the training data, it has no reason to prefer human survival to taxi-colour. It doesn't even have a reason to have a specific feature resembling "passenger survived" at all.

This might be improved by the "allow decorrelation" approach from section 4.10: we specifically allow it to maximise speed of transport, while keeping the other features (no accidents, no speeding tickets) intact. As in section 6.7, we'll attempt to check that the AI does prioritise human survival, and that it will warn us if a refactoring moves it away from this.

  1. Now, sometimes worlds w1,w2∈W may be indistinguishable for any feature set. But in that case, they can't be distinguished by any observations, either, so their relative probabilities won't change: as long as it's defined, P(w1|o)/P(w2|o) is constant for all observations o. So we can replace w1 and w2 with {w1,w2}, of prior probability P({w1,w2})=P(w1)+P(w2). Doing this for all indistinguishable worlds (which form an equivalence class) gives W′, a set of distinguishable worlds, with a well defined P on it. ↩︎

  2. It's useful to contrast a refinement with the "abstraction" defined in this sequence. An abstraction throws away irrelevant information, so is not generally a refinement. Sometimes they are exact opposites, as the ideal gas law is an abstraction of the movement of all the gas particles, while the opposite would be a refinement.

    But they are exact opposites either. Starting with the neurons of the brain, you might abstract them to "emotional states of mind", while a refinement could also add "emotional states of mind" as new features (while also keeping the old features). A splintering is more the opposite of an abstraction, as it signals that the old abstraction features are not sufficient.

    It would be interesting to explore some of the concepts in this post with a mixture of refinements (to get the features we need) and abstractions (to simplify the models and get rid of the features we don't need), but that is beyond the scope of this current, already over-long, post. ↩︎

  3. Specifically, we'd point - via labelled examples - at a clusters of features that correlate with functioning digestion, and another cluster of features that correlate with brain activity, and allow those two clusters to decorrelate with each other. ↩︎

  4. It is no coincidence that, if R and R′ are rewards on M, that are identical on E, and if R∗ is a refactoring of R, then R∗ is also a refactoring of R′. ↩︎

  5. Though note there are some problems with this approach, both in theory and in practice. ↩︎

  6. Some more "body instincts" skills require more realistic environments, but some skills and procedures can perfectly well be trained in minimal simulators. ↩︎

  7. You could define honour as "behaves according to the implicit expectations of their society", but that just illustrates how time-and-place dependent honour is. ↩︎

  8. Pre 1870. ↩︎

  9. It's not impossible to get superhuman performance from apprenticeship learning; for example, we could select the best human performance on a collection of distinct tasks, and thus get the algorithm to have a overall performance that no human could ever match. Indeed, one of the purposes of task decomposition is to decompose complex tasks in ways that allow apprenticeship-like learning to have safe and very superhuman performance on the whole task. ↩︎



Discuss

Are We Right about How Effective Mockery Is?

Новости LessWrong.com - 27 августа, 2020 - 13:48
Published on August 27, 2020 10:48 AM GMT

Crossposted from Figuring Figuring.

I did a second survey that fixed some of the flaws of the first survey. The results from the second survey significantly color the interpretation of the results from the first survey given in the first “Conclusion and Discussion” section. Please continue reading past the section titled “Second Survey” to get a full picture of the results from all surveys.

Intro

A couple days ago a friend of mine on facebook asked about arguments in favor of mockery. They pointed out that they had noticed a lot of facebook posts mocking people for not wearing masks in the covid-19 era, and wondered whether this was an effective way to change people’s behaviors.

I said in the comment section of that post that I would make a survey that worked as follows. Roughly half of the survey takers would be randomly assigned to answer the following questions:

  1. Do you think that mockery is an effective way to change people’s minds?
  2. Do you think that mockery is an effective way to change people’s behaviors?

The other half would be randomly assigned to answer these questions:

  1. Has being mocked ever caused you to change your mind about something?
  2. Has being mocked ever caused you to change your habits or behaviors?

No survey respondent was permitted to see all four questions. The possible answers to each question were “Yes”, “No”, and “Not sure”.

I made this survey using GuidedTrack. I posted it on my facebook wall, and also posted it to Positly and paid people to participate.

A total of 145 people responded to any of the questions on positly. 74 were asked the first set of questions, and 71 were asked the second set of questions. A total of 66 people responded to any of the questions on facebook. 31 were asked the first set of questions, and 35 were asked the second set of questions.

Before I go on to tell you the results of the survey and the predictions me and some of my friends made, you might want to make your own predictions. I suggest you quickly scribble them down. Some particular questions you might want to make predictions about:

  • Did more people answer yes to the first set of questions than the second set of questions, or is the reverse true, or were they about the same?
  • Were facebook respondents (presumably people who are friends, or friends of friends of mind on facebook) more or less likely to say yes to the first set of questions?
  • Were facebook respondents more or less likely to say yes to the second set of questions?
  • What did I predict about the previous two questions?

There may be other fun questions to predict, and I’d be curious to hear how you did in the comments. Predictions from me and my friends coming up, so make sure you make your predictions beforehand. Again, I suggest that you write them down. You may also want to write down your reasoning beforehand.

_
_
_
_
_
_
_
_
_
__
_
_
_
_
_
_
____
__
_
_
__
_
_____
_
_
_
_
_
_
_
_
_
_
_
_
____
_
_
_
_
_

Predictions

Ok, last chance to make predictions before you hear some spoilers…

Alright.

I predicted that many more people would answer yes to the first set of questions (ie, the questions about whether mockery is effective) than to the second set of questions. I also predicted more people would say no to the second set of questions than to the first.

I’m not sure exactly what my theory was when I made that prediction—I made the prediction in the same comment that I suggested the survey, but I came up with two post hoc hypotheses that might explain the result I predicted. I do know that part of the reason I made that prediction is that mockery is fun, but admitting that fun is the main reason we do it rather than because of its positive effects on other people’s behavior feels kind of icky. So we use its effectiveness as an excuse.

One hypothesis is that we overestimate the effectiveness of mockery. This would make sense of the predicted result because it would be evidence that we all think mockery works on others, but none of us thinks it works on us.

The second hypothesis I made up to explain this predicted result was that while we know that mockery works on other people, we are hesitant to admit that it works on us, because that is a bit embarrassing. Perhaps people are also not that great at telling what actually caused them to change their minds or behaviors.

These hypotheses are not mutually exclusive.

In a private conversation, my friend predicted using similar reasoning that actually people would tend to answer the second set of questions (ie, those about how often we change our own minds as a result of mockery) affirmatively. Saying that you think mockery is effective feels kind of icky, but saying that you think you have never had your mind or behavior changed because of mockery seems kind of arrogant.

Seeing how such similar reasoning could be used to predict a totally different result made me feel a bit nervous.

Another friend of mine predicted that my facebook friends would be less likely to change their minds because of mockery than randomly selected survey participants. Positly users aren’t quite randomly selected, but they’re closer to randomly selected than whatever people happened to come across my facebook post.

Results

Sure enough, I was totally wrong.

Twice as many Positly respondents said that mockery has worked on them as said that mockery is effective. Positly respondents were slightly more likely to say that mockery is effective for changing behavior than for changing people’s minds, both for themselves and for others.

I think this is strong evidence against the hypotheses I suggested, and some evidence in favor of the hypothesis my friend suggested in conversation.

My facebook acquaintances were slightly less disproportionate. Only 1.2 times as many respondents said that mockery is effective on themselves as said that mockery is effective on others when it comes to behavior. However, when it comes to changing minds, still about twice as many said that mockery has worked on them as said that mockery is effective on others.

I was surprised by this, as I tend to think of myself as preferring people who do not use mockery, and not using mockery while also thinking it is effective is a hard pair of things for humans to do simultaneously.

Of the 35 facebook respondents that were asked the second set of questions, 37% said they had changed their mind because of mockery, 59% of Positly respondents said the same. This seems like decent evidence to me that my friend was right about my facebook acquaintances being less likely to change their minds because of mockery.

No respondent said that mockery was effective for changing their own, or other people’s minds and also answered that it was not effective for changing their own, or other people’s behaviors.

Other Responses

Here is a list of some of the things that people said they changed because of mockery. This was an optional part of the survey. Slightly edited for brevity, to protect anonymity, and avoid repeats.

  • Economic beliefs
  • Working out habits
  • Avoiding people who mock them
  • Writing about topics on fb
  • My own appearance
  • How good other people are
  • Picking my nose in public
  • Crying in public
  • Basic cultural rules, like where to sit, how to join a conversation, etc.
  • Philosophical or ethical beliefs
  • Individualism as an ethical stance
  • Fashion
  • Music
  • Using an old fashioned word
  • Beliefs about what is socially acceptable
  • Conversational habits
  • Wearing briefs instead of boxers
  • Stopped whistling
  • Stopped/started wearing shorts
  • Eating habits
  • Lost weight
  • Stopped playing sports
  • Being late
  • Hairstyle
  • Stopped watching anime
  • Mocked for being autistic, so changed the way I interact with people.
  • Started wearing make up.
  • Mocked for being outgoing, became less outgoing and confident.
  • Started liking Trump
  • Left Mormon religion
  • Started thinking more before speaking
  • Started brushing teeth more
  • Stopped being conservative

Here is a list of some of the things that people said mockery was effective for changing in other people. This was also an optional part of the survey. The entries in this list have been edited as in the previous list.

  • Weird opinions
  • The way people think
  • The way others dress
  • Haircuts
  • Mask wearing
  • How often someone complains
  • Making someone hide their opinions
  • Arrogance
  • Getting people to stop doing things around you
  • Getting someone to stop writing things in public

I think these lists are similar enough in content to rule out another explanation of this data. You might have thought that people think mocking people is an effective way to get other people to change certain kinds of things, but when they think about what sorts of things they have changed themselves because of mockery, the two categories do not have much of an intersection. These lists make that seem unlikely to me.

Discussion and Conclusion (1)

These results seem like some evidence to me that people in general underestimate the effectiveness of mockery for getting others to change their minds. This is of course not necessarily an argument for using more mockery. I, for one, take the results of this survey to be a further reason that we should not mock people.

If you thought mockery was just some harmless fun you can have with your in group, as I sort of did, you might have thought that the costs to those being mocked are actually not that great. But it seems like mockery can make someone leave their religion, stop writing in public, change their political preferences, etc. I would strongly prefer for people to make decisions about those sorts of things using object level reasoning rather than reasoning about what will cause them to be mocked less. I will now much more than before see mockery as deliberate enemy action designed to interrupt other people’s cognition—not something to be taken as a joke, especially not in the context of conversations about important topics.

Second Survey

This section was written after getting the data for my second survey which was inspired by some criticisms of the questions in the first. Everything above was written before getting that data.

On the other hand, the questions I asked people in the original survey were not exactly analogous to each other. Firstly, people might have answered the first set of questions considering that although mockery is rarely effective for changing the behaviors or beliefs of those being mocked, it might work on bystanders who watch the mocking happen. Secondly, people answering the second set of questions with a “yes” might be thinking that “yes, mockery has ever caused me to change my mind” but that does not mean it is very effective.

To correct for this, I made a second survey. Half of respondents were asked the following two questions:

  1. How often has mockery worked as a means of getting you to change your mind about something?
  2. How often has mockery worked as a means of getting you to change your habits or behavior?

The other half were asked the following two questions:

  1. How often does mockery work as a means of getting someone to change their mind about something?
  2. How often does mockery work as a means of getting someone to change their habits or behavior?

The possible responses were: “very often” , “often”, “sometimes”, “rarely”, and “almost never”.

The survey was published on positly.

I will give you some room to make predictions before showing the results.

_
_
_
___
_
_
__
_
_
_
_
_
___
_
_
___
_
____
_
___
_
__
_
_
_
___
_
_
_
_
_
____
_
____
_
_
_
_
_
_
___
_
___
___
__
_
_
_
_
_
_
_

Second Survey Results

There were a total of 115 respondents. 57 were asked the first set of questions, 58 were asked the second set of questions. Here are the results compared across groups.

Mapping “very often” to 4, “often” to 3, “sometimes” to 2, “rarely” to 1, and “almost never” to 0, this gives a mean response for group 1 question 1 of 1.4562, and a mean response for group 2 question 1 of 1.5345, meaning that respondents overall thought that mockery was slightly more effective on others than on themselves.

Using the same mapping, the average response for group 1 question 2 was 1.5789, and the average response for group 2 was 1.6667. Again, respondents overall thought that mockery was slightly more effective on others than on themselves.

Discussion and Conclusion(2)

These results contradict my original interpretation of the first survey’s data. The second survey suggests that people are in general pretty well calibrated about the effectiveness of mockery, or perhaps slightly underestimate it. I conclude that much of the effect observed in the results of survey one’s data was caused by the two effects discussed at the beginning of the “Second Survey” section and not the result of people genuinely underestimating the effectiveness of mockery.

However, I think I am still going to take mockery more seriously than I did before, mostly because I still think this survey showed me that mockery is more effective than I thought it was. The list of personal examples people gave were fairly chilling. I also imagine people cave to mockery a lot more than they are able to notice or willing to admit on a survey. Furthermore, I don’t think it was a coincidence that it was mostly me and my weirdest friends who (incorrectly) predicted that people would say that mockery is much more effective on others than it is on themselves. Probably, we weirdos have grown numb to mockery’s sting, and fallen out of touch with what it feels like to be mocked for most people.

I Would like to Thank

Ozzie Gooen for inspiring me to make these surveys with his facebook post.

Frank Bellamy and Julia Kris Dzweiria for pointing out the assymetry of the questions in the original survey.

Beth Kessler and Aryeh Englander for useful discussion.

And Spencer Greenberg as well as the whole of the Positly and Guidedtrack teams for making it much easier to run surveys like these.



Discuss

Introduction To The Infra-Bayesianism Sequence

Новости LessWrong.com - 27 августа, 2020 - 11:02
Published on August 27, 2020 8:02 AM GMT

.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}

Prelude:

Diffractor and Vanessa proudly present: The thing we've been working on for the past five months. I initially decided that Vanessa's scattered posts about incomplete models were interesting, and could benefit from being written up in a short centralized post. But as we dug into the mathematical details, it turned out it didn't really work, and then Vanessa ran across the true mathematical thing (which had previous ideas as special cases) and scope creep happened. 

This now looks like a new, large, and unusually tractable vein of research. Accordingly, this sequence supersedes all previous posts about incomplete models, and by now we've managed to get quite a few interesting results, and have ideas for several new research directions.

Diffractor typed everything up and fleshed out the proof sketches, Vanessa originated almost all of the ideas and theorems. It was a true joint effort, this sequence would not exist if either of us were absent. Alex Mennen provided feedback on drafts to make it much more comprehensible than it would otherwise be, and Turntrout and John Maxwell also helped a bit in editing.

Be aware this sequence of posts has the math textbook issue where it requires loading a tower of novel concepts that build on each other into your head, and cannot be read in a single sitting. We will be doing a group readthrough on MIRIxDiscord where we can answer questions and hopefully get collaborators, PM me to get a link.

TLDR: Infra-Bayesianism is a new approach to epistemology / decision theory / reinforcement learning theory, which builds on "imprecise probability" to solve the problem of prior misspecification / grain-of-truth / nonrealizability which plagues Bayesianism and Bayesian reinforcement learning. Infra-Bayesianism also naturally leads to an implementation of UDT, and (more speculatively at this stage) has applications to multi-agent theory, embedded agency and reflection. This post is the first in a sequence which lays down the foundation of the approach.

 

Introduction:

Learning theory traditionally deals with two kinds of setting: "realizable" and "agnostic" or "non-realizable". In realizable settings, we assume that the environment can be described perfectly by a hypothesis inside our hypothesis space. (AIXI is an example of this) We then expect the algorithm to converge to acting as if it already knew the correct hypothesis. In non-realizable settings, we make no such assumption. We then expect the algorithm to converge to the best approximation of the true environment within the available hypothesis space.

As long as the computational complexity of the environment is greater than the computational complexity of the learning algorithm, the algorithm cannot use an easy-to-compute hypothesis that would describe the environment perfectly, so we are in the nonrealizable setting. When we discuss AGI, this is necessarily the case, since the environment is the entire world: a world that, in particular, contains the agent itself and can support other agents that are even more complex, much like how halting oracles (which you need to run Solomonoff Induction) are nowhere in the hypotheses which Solomonoff considers. Therefore, the realizable setting is usually only a toy model. So, instead of seeking guarantees of good behavior assuming the environment is easy to compute, we'd like to get good behavior simply assuming that the environment has some easy-to-compute properties that can be exploited.

For offline and online learning there are classical results in the non-realizable setting, in particular VC theory naturally extends to the non-realizable setting. However, for reinforcement learning there are few analogous results. Even for passive Bayesian inference, the best non-realizable result found in our literature search is Shalizi's which relies on ergodicity assumptions about the true environment. Since reinforcement learning is the relevant setting for AGI and alignment theory, this poses a problem.

Logical inductors operate in the nonrealizable setting, and the general reformulation of them in Forecasting Using Incomplete Models is of interest for broader lessons applicable to acting in an unknown environment. In said paper, reality can be drawn from any point in the space of probability distributions over infinite sequences of observations, Δ(Oω). Almost all of the points in this space aren't computable, and because of that, we shouldn't expect convergence to the true environment, as occurs in the realizable setting where the true environment lies in your hypothesis space.

However, even if we can't hope to learn the true environment, we can at least hope to learn some property of the true environment, like "every other bit is a 0", and have our predictions reflect that if it holds. A hypothesis in this setting is a closed convex subset of Δ(Oω) which can be thought of as "I don't know what the true environment is, but it lies within this set". The result obtained in the above-linked paper was, if we fix a countable family of properties that reality may satisfy, and define the inductor based on them, then for all of those which reality fulfills, the predictions of the inductor converge to that closed convex set and so fulfill the property in the limit.

 

What About Environments?

However, this just involves sequence prediction. Ideally, we'd want some space that corresponds to environments that you can interact with, instead of an environment that just outputs bits. And then, given a suitable set B in it... Well, we don't have a fixed environment to play against. The environment could be anything, even a worst-case one within B. We have Knightian uncertainty over our set of environments, it is not a probability distribution over environments. So, we might as well go with the maximin policy.

argmaxπinfe∈B(Eπ⋅e[U])

Where π⋅e is the distribution over histories produced by policy π interacting with environment e. U is just some utility function.

When we refer to "Murphy", this is referring to whatever force is picking the worst-case environment to be interacting with. Of course, if you aren't playing against an adversary, you'll do better than the worst-case utility that you're guaranteed. Any provable guarantees come in the form of establishing lower bounds on expected utility if a policy is selected.

The problem of generating a suitable space of environments was solved in Reinforcement Learning With Imperceptible Rewards. If two environments are indistinguishable by any policy they are identified, a mixture of environments corresponds to picking one of the component environments with the appropriate probability at the start of time, and there was a notion of update.

However, this isn't good enough. We could find no good update rule for a set of environments, we had to go further.

 

Which desiderata should be fulfilled to make maximin policy selection over a set of environments (actually, we'll have to generalize further than this) to work successfully? We'll have three starting desiderata.

Desideratum 1: There should be a sensible notion of what it means to update a set of environments or a set of distributions, which should also give us dynamic consistency. Starting with a policy π, a policy π′ which acts similarly to π but past history h acts better (according to the beliefs of the agent after observing h) than the usual behavior of π after h, should do better as viewed from the start.

 

Desideratum 2: Our notion of a hypothesis (set of environments) in this setting should collapse "secretly equivalent" sets, such that any two distinct hypotheses behave differently in some relevant aspect. This will require formalizing what it means for two sets to be "meaningfully different", finding a canonical form for an equivalence class of sets that "behave the same in all relevant ways", and then proving some theorem that says we got everything.

 

Desideratum 3: We should be able to formalize the "Nirvana trick" (elaborated below) and cram any UDT problem where the environment cares about what you would do, into this setting. The problem is that we're just dealing with sets of environments which only depend on what you do, not what your policy is, which hampers our ability to capture policy-dependent problems in this framework. However, since Murphy looks at your policy and then picks which environment you're in, there is an acausal channel available for the choice of policy to influence which environment you end up in.

The "Nirvana trick" is as follows. Consider a policy-dependent environment, a function Π×(A×O)<ω×A→ΔO (Ie, the probability distribution over the next observation depends on the history so far, the action you selected, and your policy). We can encode a policy-dependent environment as a set of policy-independent environments that don't care about your policy, by hard-coding every possible deterministic policy into the policy slot, making a family of functions of type (A×O)<ω×A→ΔO, which is the type of policy-independent environments. It's similar to taking a function f(x,y), and plugging in all possible x to get a family of functions that only depend on y.

Also, we will impose a rule that, if your action ever violates what the hard-coded policy predicts you do, you attain Nirvana (a state of high or infinite reward). Then, Murphy, when given this set of environments, will go "it'd be bad if they got high or infinite reward, thus I need to pick an environment where the hard-coded policy matches their actual policy". When playing against Murphy, you'll act like you're selecting a policy for an environment that does pay attention to what policy you pick. As-stated, this doesn't quite work, but it can be repaired.

There's two options. One is making Nirvana count as infinite reward. We will advance this to a point where we can capture any UDT/policy-selection problem, at the cost of some mathematical ugliness. The other option is making Nirvana count as 1 reward forever afterward, which makes things more elegant, and it is much more closely tied to learning theory, but that comes at the cost of only capturing a smaller (but still fairly broad) class of decision-theory problems. We will defer developing that avenue further until a later post.

 

A Digression on Deterministic Policies

We'll be using deterministic policies throughout. The reason for using deterministic policies instead of probabilistic policies (despite the latter being a larger class), is that the Nirvana trick (with infinite reward) doesn't work with probabilistic policies. Also, probabilistic policies don't interact well with embeddedness, because it implicitly assumes that you have a source of random bits that the rest of the environment can never interact with (except via your induced action) or observe.

Deterministic policies can emulate probabilistic policies by viewing probabilistic choice as deterministically choosing a finite bitstring to enter into a random number generator (RNG) in the environment, and then you get some bits back and act accordingly.

However, we aren't assuming that the RNG is a good one. It could be insecure or biased or nonexistent. Thus, we can model cases like Death In Damascus or Absent-Minded Driver where you left your trusty coin at home and don't trust yourself to randomize effectively. Or a nanobot that's too small to have a high bitrate RNG in it, so it uses a fast insecure PRNG (pseudorandom number generator). Or game theory against a mindreader that can't see your RNG, just the probability distribution over actions you're using the RNG to select from, like an ideal CDT opponent. It can also handle cases where plugging certain numbers into your RNG chip cause lots of heat to be released, or maybe the RNG is biased towards outputting 0's in strong magnetic fields. Assuming you have a source of true randomness that the environment can't read isn't general enough!

 

Motivating Sa-Measures

Sets of probability distributions or environments aren't enough, we need to add in some extra data. This can be best motivated by thinking about how updates should work in order to get dynamic consistency.

Throughout, we'll be using a two-step view of updating, where first, we chop down the measures accordingly (the "raw update"), and then we renormalize back up to 1.

So, let's say we have a set of two probability distributions μ1 and μ2. We have Knightian uncertainty within this set, we genuinely don't know which one will be selected, it may even be adversarial. μ1 says observation o has 0.5 probability, μ2 says observation o has 0.01 probability. And then you see observation o! The wrong way to update would be to go "well, both probability distributions are consistent with observed data, I guess I'll update them individually and resume being completely uncertain about which one I'm in", you don't want to ignore that one of them assigns 50x higher probability to the thing you just saw.

However, neglecting renormalization, we can do the "raw update" to each of them individually, and get m1 and m2 (finite measures, not probability distributions), where m1 has 0.5 measure and m2 has 0.01 measure.

Ok, so instead of a set of probability distributions, since that's insufficient for updates, let's consider a set of measures m, instead. Each individual measure in that set can be viewed as λμ, where μ is a probability distribution, and λ≥0 is a scaling term. Note that λ is not uniform across your set, it varies depending on which point you're looking at.

 

However, this still isn't enough. Let's look at a toy example for how to design updating to get dynamic consistency. We'll see we need to add one more piece of data. Consider two environments where a fair coin is flipped, you see it and then say "heads" or "tails", and then you get some reward. The COPY Environment gives you 0 reward if you say something different than what the coin shows, and 1 reward if you match it. The REVERSE HEADS Environment always you 0.5 reward if the coin comes up tails, but it comes up heads, saying "tails" gets you 1 reward and "heads" gets you 0 reward. We have Knightian uncertainty between the two environments.

For finding the optimal policy, we can observe that saying "tails" when the coin is tails helps out in COPY and doesn't harm you in REVERSE HEADS, so that's a component of an optimal policy.

Saying "tails" no matter what the coin shows means you get 0.5⋅0+0.5⋅1=0.5 utility on COPY, and 0.5⋅1+0.5⋅0.5=0.75 utility on REVERSE HEADS. Saying "tails" when the coin is tails and "heads" when the coin is heads means you get 0.5⋅1+0.5⋅1=1 utility on COPY and 0.5⋅0+0.5⋅0.5=0.25 utility on REVERSE HEADS. Saying "tails" no matter what has a better worst-case value, so it's the optimal maximin policy.

Now, if we see the coin come up heads, how should we update? The wrong way to do it would be to go "well, both environments are equally likely to give this observation, so I've got Knightian uncertainty re: whether saying heads or tails gives me 1 or 0 utility, both options look equally good". This is because, according to past-you, regardless of what you did upon seeing the coin come up "tails", the maximin expected values of saying "heads" when the coin comes up heads, and saying "tails" when the coin comes up heads, are unequal. Past-you is yelling at you from the sidelines not to just shrug and view the two options as equally good.

Well, let's say you already know that you would say "tails" when the coin comes up tails and are trying to figure out what to do now that the coin came up heads. The proper way to reason through it is going "I have Knightian uncertainty between COPY which has 0.5 expected utility assured off-history since I say "tails" on tails, and REVERSE HEADS, which has 0.25 expected utility assured off-history. Saying "heads" now that I see the coin on heads would get me (0.5×1)+0.5=1 expected utility in COPY and (0.5×0)+0.25=0.25 utility in REVERSE HEADS, saying "tails" would get me (0.5×0)+0.5=0.5 utility in COPY and (0.5×1)+0.25=0.75 utility in REVERSE HEADS, I get higher worst-case value by saying "tails"." And then you agree with your past self re: how good the various decisions are.

Huh, the proper way of doing this update to get dynamic consistency requires keeping track of the fragment of expected utility we get off-history.

Similarly, if you messed up and precommitted to saying "heads" when the coin comes up tails (a bad move), we can run through a similar analysis and show that keeping track of the expected utility off-history leads you to take the action that past-you would advise, after seeing the coin come up heads.

 

So, with the need to keep track of that fragment of expected utility off-history to get dynamic consistency, it isn't enough to deal with finite measures m, that still isn't keeping track of the information we need. What we need is (m,b), where m is a finite measure, and b is a number ≥0. That b term keeps track of the expected value off-history so we make the right decision after updating. (We're glossing over the distinction between probability distributions and environments here, but it's inessential)

We will call such a (m,b) pair an "affine measure", or "a-measure" for short. The reason for this terminology is because a measure can be thought of as a linear function from the space of continuous functions to R. But then there's this +b term stuck on that acts as utility, and a linear function plus a constant is an affine function. So, that's an a-measure. A pair of a finite measure and a b term where b≥0.

 

But wait, we can go even further! Let's say our utility function of interest is bounded. Then we can do a scale-and-shift until it's in [0,1].

Since our utility function is bounded in [0,1]... what would happen if you let in measures with negative parts, but only if they're paired with a sufficiently large b term? Such a thing is called an sa-measure, for signed affine measure. It's a pair of a finite signed measure and a b term that's as-large-or-larger than the amount of negative measure present. No matter your utility function, even if it assigns 0 reward to outcomes with positive measure and 1 reward to outcomes with negative measure, you're still assured nonnegative expected value because of that +b term. It turns out we actually do need to expand in this direction to keep track of equivalence between sets of a-measures, get a good tie-in with convex analysis because signed measures are dual to continuous functions, and have elegant formulations of concepts like minimal points and the upper completion.

Negative measures may be a bit odd, but as we'll eventually see, we can ignore them and they only show up in intermediate steps, not final results, much like negative probabilities in quantum mechanics. And if negative measures ever become relevant for an application, it's effortless to include them.

 

Belief Function Motivation

Also, we'll have to drop the framework we set up at the beginning where we're considering sets of environments, because working with sets of environments has redundant information. As an example, consider two environments where you pick one of two actions, and get one of two outcomes. In environment e0, regardless of action, you get outcome 0. In environment e1, regardless of action, you get outcome 1. Then, we should be able to freely add an environment e2, where action 0 implies outcome 0, and where action 1 implies outcome 1. Why?

Well, if your policy is to take action 0, e2 and e0 behave identically. And if your policy is to take action 1, e2 and e1 behave identically. So, adding an environment like this doesn't affect anything, because it's a "chameleon environment" that will perfectly mimic some preexisting environment regardless of which policy you select. However, if you consider the function mapping an action to the set of possible probability distributions over outcomes, adding e2 didn't change that at all. Put another way, if it's impossible to distinguish in any way whether an environment was added to a set of environments because no matter what you do it mimics a preexisting environment, we might as well add it, and seek some alternate formulation instead of "set of environments" that doesn't have the unobservable degrees of freedom in it.

To eliminate this redundancy, the true thing we should be looking at isn't a set of environments, but the "belief function" from policies to sets of probability distributions over histories. This is the function produced by having a policy interact with your set of environments and plotting the probability distributions you could get. Given certain conditions on a belief function, it is possible to recover a set of environments from it, but belief functions are more fundamental. We'll provide tools for taking a wide range of belief functions and turning them into sets of environments, if it is desired.

Well, actually, from our previous discussion, sets of probability distributions are insufficient, we need a function from policies to sets of sa-measures. But that's material for later.

 

Conclusion

So, our fundamental mathematical object that we're studying to get a good link to decision theory is not sets of probability distributions, but sets of sa-measures. And instead of sets of environments, we have functions from policies to sets of sa-measures over histories. This is because probability distributions alone aren't flexible enough for the sort of updating we need to get dynamic consistency, and in addition to this issue, sets of environments have the problem where adding a new environment to your set can be undetectable in any way.

 

In the next post, we build up the basic mathematical details of the setting, until we get to a duality theorem that reveals a tight parallel between sets of sa-measures fulfilling certain special properties, and probability distributions, allowing us to take the first steps towards building up a version of probability theory fit for dealing with nonrealizability. There are analogues of expectation values, updates, renormalizing back to 1, priors, Bayes' Theorem, Markov kernels, and more. We use the "infra" prefix to refer to this setting. An infradistribution is the analogue of a probability distribution. An infrakernel is the analogue of a Markov kernel. And so on.

 

The post after that consists of extensive work on belief functions and the Nirvana trick to get the decision-theory tie-ins, such as UDT behavior while still having an update rule, and the update rule is dynamically consistent. Other components of that section include being able to specify your entire belief function with only part of its data, and developing the concept of Causal, Pseudocausal, and Acausal hypotheses. We show that you can encode almost any belief function as an Acausal hypothesis, and you can translate Pseudocausal and Acausal hypotheses to Causal ones by adding Nirvana appropriately (kinda). And Causal hypotheses correspond to actual sets of environments (kinda). Further, we can mix belief functions to make a prior, and there's an analogue of Bayes for updating a mix of belief functions. We cap it off by showing that the starting concepts of learning theory work appropriately, and show our setting's version of the Complete Class Theorem.

 

Later posts (not written yet) will be about the "1 reward forever" variant of Nirvana and InfraPOMDP's, developing inframeasure theory more, applications to various areas of alignment research, the internal logic which infradistributions are models of, unrealizable bandits, game theory, attempting to apply this to other areas of alignment research, and... look, we've got a lot of areas to work on, alright? 

If you've got the relevant math skills, as previously mentioned, you should PM me or Turntrout to get a link to the MIRIxDiscord server and participate in the group readthrough, and you're more likely than usual to be able to contribute to advancing research further, there's a lot of shovel-ready work available.

 

Links to Further Posts:

Discuss

Страницы

Подписка на LessWrong на русском сбор новостей