Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 36 минут 34 секунды назад

What would a 10T Chinchilla cost?

3 мая, 2022 - 17:48
Published on May 3, 2022 2:48 PM GMT

I've heard several people say that a 10 Trillion parameter GPT-3-like model, trained with DeepMind's new scaling laws in mind, would be pretty terrifying. I'm curious if anyone could give me a Fermi estimate of the cost of such a thing - if indeed it is feasible at all right now even with an enormous budget. 



Discuss

Does the “ugh field” phenomenon sometimes occur strongly enough to affect immediate sensory processing?

3 мая, 2022 - 16:42
Published on May 3, 2022 1:42 PM GMT

Related to my comment on the parent question: is there documentation of specific attention minimization and/or blotting-out effects in immediate sensory processing related to past emotional aversion? I suspect the two of those, despite being listed as separate bullet points in the parent question, should be treated separately…

A more generalized form of this seems like it'd be the kind of dissociation that can occur in e.g. PTSD. Do some PTSD sufferers have sharper sensory issues surrounding the brain refusing to recognize certain stimuli?



Discuss

Why humans don’t learn to not recognize danger?

3 мая, 2022 - 15:53
Published on May 3, 2022 12:53 PM GMT

Very short version in the title. A bit longer version at the end. Most of the question is context.

Long version / context:

This is something I vaguely remember reading (I think on ACX). I want to check if I remember correctly/ where I could learn it in more technical detail.

Say you go camping in a desert. You wake up and notice something that might be a scary spider you take a look and confirm it's a scary spider indeed. This is bad, you feel bad.

Since this is bad, you will be less likely to do some things that led to you will be less likely to do things led to you feeling bad, for example you'll be less likely to go camping in a desert.

But you probably won't learn to:

  • avoid looking at something that might be a scary spider or
  • stop recognizing spiders

even though those were much closer to you feeling bad (about being close to a scary spider).

This is a bit weird if you think that humans learn to just get a reward usually you'd expect stuff that happened closer to the punishment to get punished more, not less.

What I recall is that there is a different reward for "epistemic" tasks. Based on accuracy or saliency of things it recognizes, not on whether it's positive / negative.

A bit longer version of the question:

Why don't humans learn to not recognize unpleasant things (too much)? Is there a different reward for some "epistemic" processes? Where could I learn more about this?



Discuss

What would be the impact of cheap energy and storage?

3 мая, 2022 - 08:20
Published on May 3, 2022 5:20 AM GMT

Imagine fusion technology developed such that the marginal price of an additional unit of energy was ten thousand times cheaper than it is currently.

Further suppose that we invented cheap, safe, lightweight batteries with effectively unlimited storage.

What impact would that have on technology and society?



Discuss

Monthly Shorts 4/2022

3 мая, 2022 - 07:10
Published on May 3, 2022 4:10 AM GMT

Conflict

Elon Musk was once asked about the regulatory situation of providing satellite internet without the local country’s permission. His response was uniquely Muskian:

Elon Musk @elonmusk@thesheetztweetz They can shake their fist at the sky

September 1st 2021

1,290 Retweets11,916 Likes

Now, it turns out, there are also other options. Dictators can, for example, launch electronic warfare measures against SpaceX’s operations. Fortunately…it turns out that SpaceX is better than the Russians and so Ukranian internet access continues.

Fun piece on military inter-service conflict (in favor), if that’s your jam.

One of the things I’ve had to grapple with, at my age, is understanding just how meaningful 9/11 is to people older than me. Two months of car crash deaths get shown on TV, and everybody goes completely mad. I go to a panel on national security work, and every single panelist and the moderator says that their inspiration to enter government service was 9/11. The Census Bureau handed over information on Arab neighborhoods to DHS (the story is more complicated than that: DHS seems to be both lying and incompetent and the Census Bureau did something both understandable and legally required, but this is the short version). We passed the Patriot Act, setting up massive denial of civil liberties by means both legal (new authorizations) and structural (empowering a type of agency that cares very little for such things at the expense of Justice and State, which do).

DHS has seized over $500 million in currency from people who didn’t follow said signage.

State and local taxation is usually regressive in America.

Code and Consequences

This request was intended to inform the implementation work. Instead, all hell broke loose.

After the 2019 CNSTAT meeting made clear that evaluators were not accounting for the biases of the published data, the Census Bureau attempted to inform stakeholders that they were not comparing their analyses to ground truth. Having nothing else to compare the data to, this information was not well received by either data users or other census advocates.

If the Census Bureau were to publish datasets that that included fractional or negative population counts, this was bound to confuse and upset external stakeholders, including lawyers and judges.

This is a surprisingly fascinating article on the introduction of differential privacy at the Census Bureau. Reading between the lines a little, the authors imply that differential privacy reduced error, and that it was seen as increasing error because it produced different results from datasets released through more statistically “damaging” privacy-preserving processes (“swapping”). It’s somewhat an examination of the hard problems involved in communicating highly technical information to a diverse group of stakeholders, many of whom believed comfortable lies that what should have been a small process improvement made impossible. It’s a story about how a small and comprehensible technical constraint (that census data consist of census reports, not merely statistical contents)

I also get to admit an error here: I hadn’t realized the extent of data manipulation in prior census reports, and the related conclusion that the cost of introducing differential privacy is much smaller than I’d thought.

Interesting reddit post as a breakdown on crypto, from a relatively sympathetic person bringing data!

Cool new technique available on Edge for beating private mode blockers.

Some of you might have thought you’d sleep well tonight. “Fun” fact: it’s possible to insert backdoors in ML models that can’t be detected by computationally bounded observers.

Typeclasses are a really nice way of doing many sorts of classes in Python!

The future of VR is very soon and very cool. Here’s Meta talking about their new experimental feature to let users fix permanent points in physical space that stick around session to session.

History

If you just want more history, here’s a timeline, with dots for every event in Wikipedia. Try clicking on them!

Nature, in its entire history, has had 115 retracted papers. 50 of them have happened in the past seven years. Retraction Watch is really cool.

Next time you fly, you can download Flyover Country in advance, and then have a guide with you to all the cool geographic features out your window! Thank you, geoscience outreach humans!



Discuss

Notes on Empathy

3 мая, 2022 - 07:06
Published on May 3, 2022 4:06 AM GMT

This post examines empathy, as part of a sequence of posts about virtues. It is mostly an exploration of what other people have learned about empathy, rather than my own research or opinions about it, though I’ve been selective about what I found interesting or credible. I wrote this not as an expert on the topic, but as someone who wants to learn more about it. I hope it will help people who want to know more about empathy and how to practice it skillfully.

This was a particularly interesting (and difficult) post to research and write. There is a wealth of literature about empathy, and I found a lot of it surprising. I feel kind of like the kid who opened up the back of dad’s watch to see how it works and now is sitting amidst a cubic yard of springs and gears wondering how to put it all back together.

What is empathy?

The word “empathy” is pretty new. It apparently was coined in 1908, and only really caught on in the last half of the 20th century.[1] (The concept itself is not so new; earlier authors sometimes deployed the word “sympathy” to cover similar experiences.)

a Google Books Ngram view

The definition of empathy is contested.[2] What sort of thing empathy is also defies agreement: is it a sense or emotion (an immediate and visceral mirroring of another person’s state), or is it more like an intellectual feat (accurately discerning another person’s viewpoint), or maybe a social skill (the ability to respond appropriately to another’s condition)?

Some authors narrowly define empathy as a sense or feeling, like “the coexperience of another’s situation”[3] or “feeling what you think others are feeling.”[4] Others apply the term to a constellation of feelings, cognitive interpretations, and responses, for example “a social and emotional skill that helps us feel and understand the emotions, circumstances, intentions, thoughts, and needs of others, such that we can offer sensitive, perceptive, and appropriate communication and support”[5] or “the art of stepping imaginatively into the shoes of another person, understanding their feelings and perspectives, and using that understanding to guide your actions.”[6]

In some descriptions of empathy it is restricted to feeling distress upon noticing someone else exhibiting distress in circumstances such as pain, fear, or loss. But in others, you might just as easily empathize happily with someone experiencing more enjoyable emotions like joy or triumph (“I’m so proud of you!”), or empathize curiously with someone who is not experiencing any strong emotion at all but whose perspective you want to explore for other reasons (“I wonder what’s on his mind.”).

If all of this weren’t confusing enough, authors will sometimes slide between different definitions without seeming to realize that they’re doing it. (I’ll probably end up doing this too.)

There is plenty of interesting speculation and research about the evolutionary pathways that led to human empathy, the neurological correlates to empathic behavior, the expression of empathy (or its building blocks) by other animal species, and the developmental stages of empathy in children. I’m mostly going to leave such stuff out of my summary of the topic here, which is going to focus on how adult humans can characteristically empathize well. One important take-away from such research, however, is that different components of empathy have different foundations in development and in the brain. So people will probably differ in which components they are better or worse at. If I find myself to be unskilled at empathy I might want to look closer at which specific component seems to be lagging. And if I try to strengthen my empathy, I may have more luck if I look for ways to strengthen individual components.

People sometimes speak of empathy as a perception that operates automatically and subconsciously—or even as a variety of emotion. Other times empathy is described as a deliberate and cognitive skill of creating and working with mental models about other people. Some models incorporate both of these possibilities, working in tandem. I will also be considering empathy as a variety of virtue, which puts it more in the deliberate-skill category (a virtue is also a characteristic skill). So I want to examine (a) whether characteristic empathy is good for you (is it part of how to flourish as a human being); (b) if so, how is empathy best practiced; and (c) what practical steps can you take to get better at it.

An applause light often accompanies the word “empathy.” Authors who write about empathy sometimes gush about other things that fall into the halo surrounding empathy—like being kind, civil, tolerant, cosmopolitan, peaceful, or gentle—seemingly without realizing they’ve changed the subject. A lot of short-form popular writing on empathy doesn’t look at it critically, but assumes that it is good, full-stop, and that everybody just needs more of it. But on closer examination empathy seems not to be a more-is-better sort of virtue, but one that follows the Aristotelian “golden mean” theory: virtuous empathy is empathy characteristically practiced at the right times, in the right ways, and to the right extent. Even Karla McLaren, the most enthusiastic empathy booster of those I read, agrees that “there’s a sweet spot with empathic skills” between too little and too much.[5]

Just about everybody empathizes. One of the more empathy-skeptical writers I read nonetheless concluded that “If there were people lacking empathy completely, we would not recognize them as people.”[3] Indeed, people seem to be “hyperempathic.” It is a little weird how eager we are to empathize, for instance when we invent gods and spirits whose moods and whims we intuit to explain natural phenomena, or when we attribute emotions and motivations to things we have no reason to believe possess any, such as animated geometric shapes:

What are the components of empathy?

There seems to be some agreement that empathy is not a simple thing, but is a composite of things. There is less agreement about what those things are, or which things are part of empathy and which ought to be carved off and assigned to something else.

One common way to divide empathy is between its affective (“experience sharing,” “emotional empathy”) and cognitive (“understanding,” “mental state attribution”) aspects.[7]

Some authors restrict the term “empathy” to the affective aspects.[8]

Others add a third category that I’ll call “behavioral empathy” (a.k.a. “empathic concern”[9] or “perceptive engagement”[5]). This concerns how, once you have felt and understood another person’s point of view, you make use of that to take some appropriate action.

Affective empathy

The affective aspects are usually thought to be temporally, developmentally, and evolutionarily primary. They begin with simple mirroring or mimicry of the current emotional state of another person. You see them startle, you get frightened too (before you know what you’re supposed to be afraid of). You see them crying and tears come to your eyes (before you know what’s wrong). You laugh along at the punchline (before you get the joke). You watch Two Girls, One Cup and find yourself gagging as if you were one of the stars.

(Inevitably, “mirror neurons” are invoked to explain some of this, so I feel obligated to mention them here. These are neurons, first discovered in macaque monkeys, that fire both when a creature performs some particular action and when they view someone else performing that same action.)

Sometimes this mirroring is called “emotional contagion,” but other authors use that phrase to refer only to a pathological empathic overreaction in which you become so overwhelmed by an empathic emotion that you lose track of where it came from and feel it as though it originated with you. (Still other authors recognize that same pathology, but give it different labels, like “empathic distress” or “personal distress.”) In any case, mirroring primes the pump for the rest of the empathic process. You begin by mimicking another person’s affect, and then, by perceiving what feelings this mirroring evokes in you, you get some insight into what feelings might be going on in the other person.

People can get tripped up at this early stage of empathy. If they find empathically-evoked emotions overwhelming or unwelcome, they may try to escape from them and to suppress the affective empathy response. For example, people who have learned aversion to certain emotions, or people who learned from bad childhood experiences that emotional extremes in others can be precursors to abuse, may react in that way. People with heightened sensitivity to emotional stimulus may also find affective empathy overwhelming. Karla McLaren theorizes that the empathy deficits measured in people with autism spectrum conditions may be caused this way.[5]

A subset of “empathic accuracy” is sometimes teased out from the rest of affective empathy.[5] When you mirrored the other person’s affect, did you hit the target? Your empathy can get off to a bad start without this sort of accuracy: They laugh nervously, you laugh uproariously; they are pleasantly surprised, you become bewildered; they express amused chagrin, you become indignant on their behalf. If certain emotions trigger things in you idiosyncratically (e.g. fear⇒panic or embarrassment⇒shame), this can also interfere with empathic accuracy.

Elizabeth Segal adds “affective mentalizing” as a subcomponent of affective empathy.[10] This additional step provides more raw material for cognitive empathy to operate on. It involves appraising another’s emotional state by pulling together additional clues like body language, facial expressions, knowledge about the other’s beliefs & situation, and context (what just happened). This process can cause you to engage in an additional cascade of mirroring behavior. You can also summon affective mentalizing deliberately, without the trigger of an initial acute emotional outburst from someone else, just by deciding to pay closer attention to someone. When you develop an emotional response to a character in a story or a daydream, for example, you are doing so by means of affective mentalizing.

Cognitive empathy

“When I condole with you for the loss of your only son, in order to enter your grief, I do not consider what I, a person of such a character and profession, should suffer, if I had a son, and if that son was unfortunately to die; but I consider what I should suffer if I was really you; and I not only change circumstances with you, but I change persons and characters. My grief, therefore, is entirely upon your account, and not in the least upon my own.” ―Adam Smith[11]

Cognitive empathy also has a few subcomponents.

“Emotion regulation” is a damper on the emotional contagion process.[5] It allows even an alarming empathic emotion to appear as a cool blue info-light on your dashboard rather than a blinking red warning. Without emotional regulation, you might let such an emotion run away with you in a way that doesn’t do you or anyone else any good.

You will sometimes hear people described as “extremely empathic” because they overreact to the emotions of others—going into hysterics at another person’s distress, for instance. But this is better thought of as poorly-calibrated empathy. Some people respond to others’ pain/distress/etc. with avoidance, anti-social behavior, aggression, or personal distress, any of which demonstrates that something is amiss here.[12]

“Self-other awareness” enables you to empathize with another person while keeping your sense of self intact. It helps you realize that you are empathizing—that you are experiencing a borrowed emotion—and that you can therefore more dispassionately decide how much you want to get wrapped up in it. Without it, you can become mistaken about the nature of your feelings and your empathy can become narcissistic (“look at all the suffering your distress is causing me”).

“Perspective taking” is when you try to imagine the other person’s experience from their point of view. The first part of this is to shift from considering how you feel about what they feel to considering how they feel. This is the difference, for example, between being happy that your child is happy because it’s more pleasant to be around a happy child and being happy that your child is happy because you share in their happiness.

(This is where “theory of mind” tends to enter the discussion, but I’ll try to avoid the temptation to go down that rabbit hole.)

There are two ways to go about this shift in perspective. The first and easiest is to imagine how you would feel if you yourself were in the other’s situation. But this can be misleading.[13] Another approach is to imagine how you would feel if you were them in their situation. That requires you to engage in some difficult and speculative modeling of what it is like to be the other person, but as a result may give you a more accurate model of them to work with.

If you know the other person and their agenda well, taking their perspective is easier. If they’re a stranger to you, you have little to go on but your model of a typical person, combined with whatever stereotypes you have about people-like-them, and whatever clues you have picked up during “affective mentalizing.” This suggests that you will be better able to empathize accurately if you have better models of people in all their rich variety: which things are peculiar to you and which are more general to humanity, in what ways people are diverse, what sorts of traits and attitudes tend to cluster in people, and so forth.

Knowledge of context and social nuance comes in handy at this stage too. What signals are people putting out? Are subtle things—like plausibly-deniable slights, back-handed complements, or damning with faint praise—coming into play? If you do not have a sophisticated understanding of social dynamics, it can be easy in some circumstances to draw the wrong conclusions about another person’s state and what triggered it.

It’s important to distinguish all of this from a colder sort of assessment: “so-and-so is in such-and-such a situation, and so I intend to feel for them in some way.” In the empathic process, you go through at least an initial phase in which you feel, not for but with them. In the cognitive phase of this process, you may begin to move from this feeling-with to deciding on an appropriate feeling-for (e.g. sympathy, compassion, pity, concern), which leads us to behavioral empathy.

Behavioral empathy

So you are experiencing empathy, you feel what you believe to be a reasonable facsimile of what someone else is feeling… now what? Part of the promise of empathy is that this insight into the other person’s headspace will help you to come up with a way to interact with them that optimally fits the situation. Perhaps they have a need you could help them to meet, or perhaps they’re about to blow their top and you should beat a hasty retreat, or perhaps they’re feeling generous and now would be a great time to ask for that favor.

Typical examples of empathy-provoked responses include compassion, kindness, care, and consolation. These are often altruistically helping behaviors prompted by another’s distress.

One theory about this is that empathy prompts us to help a person who is experiencing a distressing emotion because we find empathically sharing that emotion to also be distressing and we’d like that to stop. This theory leaves it something of a mystery why we should help rather than stop empathizing, which would be as effective for that purpose and probably simpler. Empathy seems sticky: you can turn away but you can’t get it out of your head; you could have helped and that bugs you. Experimentally, even when people are given the option of an easy, clean exit from an empathically distressing situation, they still often opt to help instead.[14]

Paul Bloom believes a better explanation is that empathy helps to trigger preexisting humane motives (or, perhaps, “virtues”). Empathy does not goad us to be compassionate in order that our compassion will indirectly relieve our empathic distress, but it makes us aware that an occasion for compassion has arisen so that we can rise to the occasion if that is our wont. “It’s not that empathy itself automatically leads to kindness. Rather, empathy has to connect to kindness that already exists. Empathy makes good people better, then, because kind people don’t like suffering, and empathy makes this suffering salient. [In contrast, i]f you made a sadist more empathic, it would just lead to a happier sadist…”[15]

Another example of an empathy-provoked response might be forgiveness, which is easier for us to give when we empathically verify the other person’s remorse. And, from the other direction, apology is more effective if it appears to come from a place of vivid awareness of the harm caused (not just from an abstract acknowledgment of wrongdoing), and you can better understand that harm if you feel it empathically.[16]

Social empathy

Elizabeth Segal would like to add “social empathy” to the list. Social empathy involves a more sophisticated understanding of large-scale social dynamics: how people’s perspectives are affected by things like class, race, culture, history, systemic barriers, and so forth. It gives you the “ability to understand people by perceiving or experiencing their life situations and as a result gain insight into structural inequalities and disparities.”[17] It seems to be in part an attempt to shoehorn sometimes-parochial modern social justice concerns into empathy, and I was skeptical about how Segal et al. went about it.

A proposed Social Empathy Index test includes questions like “I think the government needs to be a part of leveling the playing field for people from different racial groups,” “I believe adults who are poor deserve social assistance,” and “I believe that each of us should participate in political activities,” that seem to uncritically import certain contested ideas into the definition of social empathy.[18] (For example: What if I think I should give a hand to adults who are poor even if they don’t “deserve” it? Does that make me more or less socially empathic? What if I think they could probably use financial assistance more than “social assistance” whatever that is?) Another such question—“I believe government should protect the rights of minorities”—seems to me like it only potentially measures empathy for others if the answerer doesn’t consider themselves one of the minorities who would be protected (that Latinos in the U.S. have higher SEI scores than Whites shouldn’t be surprising even if you suspect both groups have a similar amount of “social empathy,” given questions like that).

In part, “social empathy” is meant to address well-documented common biases in human empathy, such as those that favor ingroups over outgroups. “How do we cultivate the type of empathic reactions that people demonstrate toward friends and family members in their responses to groups who differ from them, particularly groups that have historically been the focus of prejudice and oppression?”[19]

Other models

Fritz Breithaupt has an idiosyncratic model for empathy.[20] If I understand him right, he says that a typical person has most of the various “abilities and mechanisms related to empathy” described in the Affective, Cognitive, and Behavioral sections above, “and a [hyperempathic] tendency to use them;” but also has a vigorous set of empathy suppression mechanisms. These suppression mechanisms operate at multiple levels, from the neural, to the conscious (e.g. evaluating when someone is emotionally manipulating us; deciding when someone had-it-coming), to the societal (e.g. learning which animals we ought to feel empathy toward and which ones we can mercilessly chop up for bait). They prevent empathy from disturbing us unless certain conditions are met. In Breithaupt’s model, these suppression mechanisms are as important to well-regulated empathy as any of the positive abilities and mechanisms, but they also have biases and idiosyncrasies.

How is empathy measured?

There has been a lot of research into empathy, of a more-or-less scientific nature. Much of the challenge of such research comes from how to measure empathy. A typical approach is to give people self-assessment questionnaires in which they are asked how much they feel certain descriptions apply to them (e.g. “When someone else is feeling excited, I tend to get excited too” or “I become irritated when someone cries”).

But this means researchers are not measuring empathy itself, but how much people report that statements about empathy apply to them. This sort of indirect measurement is flawed. It is not unheard of for people to have inaccurate self-images, for example. And people may respond to such questions aspirationally (what they would like to be, or what they think they should be) rather than accurately.

One puzzled researcher found that students scored as less empathetic on such an empathy questionnaire after taking her empathy class. Why? After learning more about empathy, the students became more exacting in how they judged their own feelings and behavior, expected more from themselves, and so rated themselves more severely.[19] So someone with a low “empathy quotient” might either be someone who has little empathy or someone who has plenty but has even higher standards.

Similarly, when gender differences in empathy scores are found, it turns out to be very difficult to tease out whether this is because one gender is really more empathic than another, or whether people are just conforming to gender expectations in their self-image or in how they think it is appropriate to present themselves.

Another problem is that as the science of empathy has matured, and additional facets of empathy have been teased out or theorized by researchers, new questionnaires have been developed to try to capture these nuances. There are now multiple competing empathy questionnaires, which can make it difficult to compare results from study to study.

Even when the same questionnaire is used, it may not really measure the same thing over time because language changes. One study, which used a single questionnaire to measure 13,737 college students between 1979 and 2009, found that empathy measures had been falling substantially over that time.[21] The press of course went with the “kids these days” angle.[22] But one critic noted the results might be explained by the fact that the wording of the questionnaire had become anachronistically quaint (it used idioms no longer in common use, like “tender feelings”, “ill at ease”, “quite touched”, or “go to pieces”), and today’s students might not identify with such statements for that reason.[3]

Other ways to measure aspects of empathy include fMRIs and things of that sort, in which subjects are asked to perform empathy-related tasks while their brain activity is measured. Researchers can monitor different sorts of people (e.g. sociopaths, autistic people, children, meditators) or people in different circumstances or undergoing different interventions to see if their brains light up in different ways, for what that’s worth.

Some researchers have tried to measure empathic accuracy by filming one person, later asking that person what was going through their mind at various points during the filming, asking a second subject to watch the film and to try to empathically discern those subterranean thoughts and feelings, and then comparing that person’s answers to what the first person reported. It’s a complex experiment design that leans heavily on human memory, introspection, language-use, and subjective judgment. But it may be as close as we can expect to get to measuring and comparing the subjective and ephemeral.

Is empathy altruistic? is it moral?

Paradigmatically, empathy is a tender feeling toward someone else in crisis, followed by consolation or kind assistance. You feel someone’s pain and then you work to relieve it, for their sake. Is empathy essentially altruistic? This is also contested. It turns out to be pretty easy to tell a story in which seemingly altruistic acts turn out to be motivated primarily by self interest after all. But it is difficult to test whether this says as much about the seemingly altruistic acts as it does about our story-telling abilities.

“You say: How sad to think that the noblest altruism is, after all, merely a refined kind of selfishness.

“I say: How good to think that selfishness, when it is purified and stops being stupid, is exactly the same thing as the noblest kind of altruism.”

―Pierre Cérésole[23]

Empathy has been proposed as the primitive foundation on which humans established moral ideas (Martin Hoffman calls empathy the “bedrock of morality” for example).[24] Once empathy allowed us to see things from other people’s points of view, we then became able to entertain ideas like The Golden Rule, mutual tolerance, and so forth.

This further extends into political rights: If other people have aspirations, needs, etc. just like I do, they perhaps ought to have rights that I should respect, and vice-versa. Jeremy Rifkin for this reason called empathy the “soul of democracy.”[25]

Empathy skeptics, however, argue that because of the strong, demonstrable, and not particularly defensible biases of human empathy, it makes a poor foundation for morality or for political rights (see below for more about this). But Roman Krznaric points out that in historical accounts of societal moral progress, such as the abolition of the slave trade, successful appeals to empathy can seem to have played a crucial role. “Empathy and reason are not polar opposites, as critics like [Paul] Bloom would have us believe, but rather mutually reinforcing ideals on which we can build a more humane civilization. Indeed, it is ‘the gut wrench of empathy’ [quoting Bloom] that forces open the door of our common concern—and only then does reason have a chance to wedge it open with laws and rights.”[26] While it may be important that we use reason rather than biased empathy to make decisions about the worth of human lives, “the explanation for why we believe all humans should be treated and valued equally… is because empathy has made us care about the plight of strangers outside our local community.”[26]

Filtered empathy

Fritz Breithaupt is one of those who disagrees with the assessment that empathy is altruistic. “Like most other human abilities, empathy probably serves the empathizer first and foremost and not the target of empathy.”[3] He believes that empathy is better understood as a self-serving feeling: a mostly aesthetic indulgence and a way of satisfying our curiosity about others.

Breithaupt points out that in humanitarian empathy in particular it’s questionable whether we really empathize with those in need or rather with a (perhaps imaginary) rescuer whose hero role we empathically inhabit. Such empathy allows us to feel not the suffering of someone in crisis, but the praise due to the rescuer, and so makes empathy more alluring—we don’t feel bad for them but good for us. He calls this “filtered empathy”—empathy that is absorbed by a third party before reaching its ostensible target. (This may help to explain the common Hollywood trope of telling stories of struggling people in a roundabout way through outsiders who intervene to help them, e.g. Schindler’s List, Amistad, Avatar.)

This also may explain why we sometimes empathize with people we cannot possibly help (victims we hear about on the news, fictional characters, historical figures). Such empathy can’t culminate in helpful action, but if Breithaupt is correct, that is not its purpose: instead it culminates in a pleasant Walter Mitty-like daydream along the lines of “if only I had been there, I would have helped heroically.”

Empathic accuracy

Native Mandarin Chinese speakers overestimated how well native English-speaking Americans understood what they said in Chinese, even when they were informed that the listeners knew no Chinese. These listeners also believed they understood the intentions of the Chinese speakers much more than they actually did. This extreme illusion impacts theories of speech monitoring and may be consequential in real-life, where miscommunication is costly.[27]

When I empathize with someone, I have the sense that I really feel what they feel. Part of this comes from how visceral the emotion is: it doesn’t seem like I’m making up a story but like something has affected me. But do I have good cause to believe my empathic feelings are any more accurate than any other guesswork I might engage in about the contents of other people’s heads?

Is there even theoretically enough information available to a maximally astute observer to accurately surmise the inner state of the person observed? How transparent are we, really? People often seem to throw up clouds of ink in the waters around them, and can be mysteries even to themselves.

Even given the information that is available, just how clever can we expect imperfect mortals to be in putting it all together? Can we trust our intuition about how well we do at this complex task?

Correspondence bias a.k.a. the fundamental attribution error is one example of how we predictably mislead ourselves by observing others and then jumping to conclusions about their motives and outlooks.

Even assuming you have good, representative, unambiguous data about another person, and you use that data wisely to come up with a good model of how they work, is that anything like understanding how it feels to be them? What is it like to be a bat? I necessarily use myself as the only available model for what a subjectivity feels like, and so I’m bound to make errors if I try to generalize from my n=1. Every once in a while I learn something new, for instance that some people don’t have an inner monologue, and I have to throw out big hunks of my model of what makes other people tick. There’s lots of stuff like this, apparently.

Fritz Breithaupt is among those who think empathy is deceptive in this way. An empathizer simplifies the situation of the person they are empathizing with to what seem to be the most relevant features; to that other person, the situation is messier. “[T]he feelings of the other person become a fact to [the empathizer], appearing transparent and perceptible.”[3] This can give the empathizer “a clarity not available to the other”—a clarity that may sometimes even be helpful, if the person being empathized with is lost in a thicket of details and uncertainty.

But this oversimplification may just be true of rudimentary, naive empathy. Another author stressed that more proficient empathy doesn’t have to be so reductive. “Empathy means acknowledging a horizon of context that extends perpetually beyond what you can see… Empathy means realizing no trauma has discrete edges.”[28]

Empathic accuracy is hard, but I don’t want to just give up. Awful things can sneak in through that door. For instance, you can justify doing cruel things to others if you can convince yourself that there’s just no way of knowing whether it’s really cruel from their point of view: e.g. “the Arab mind is different from ours,” “homeless people probably just like to live that way,” or “animals probably don’t really suffer.”

Karla McLaren recommends that to improve the accuracy of your empathic sizing up of someone else, you just frickin’ ask them. “The way to gauge your Empathic Accuracy is both very simple and infinitely hard: you ask people if what you’re sensing from them is true.”[29] You’re never going to become a supernatural mind-reader. People exhibit themselves in diverse and complex ways, and to some extent each person has to be interpreted anew as if they were written in a foreign language. Your guesses will be prone to errors, but you can always go to the expert: the person whose feelings you are trying to discern. McLaren recommends that you ask like this:[30]

“When you [objectively described behavior], it seems to me that you [want/feel x]. Did I get that right?”

For example:

“When you cross your arms like that, it seems to me that you feel impatient. Did I get that right?”

(She stresses that it’s important that you not import your guesses about their feelings or inner state into your description of their behavior. So, for instance “when you gaze out the window…” is okay, but “when you start ignoring me…” is less helpful.)

When we learn that our empathic assessments are incorrect, we can recalibrate and bring the other person into better focus. Unfortunately, according to Fritz Breithaupt, “false empathy is a powerful drug” and it can be difficult for people to abandon bad guesses once they’re empathically established.[3] Sometimes people respond to being told their empathic accuracy is off by accusing the person they are empathizing with of not knowing their own mind (this isn’t to say all such accusations are false, but it does take some chutzpah to assert that I know your feelings better than you do), or by getting angry at them for not feeling the way they’re supposed to or for not being sufficiently legible.

Relation to other virtues

Empathy can contribute to other-attending virtues like compassion, kindness, care, sympathy, pity, consideration, courtesy, nying je, consolation, altruism, recognition, respect for others, persuasion/education/tutoring, loyalty, amiability, and connection.

It can be helped by mindfulness/attention, emotional intelligence, curiosity, and imagination. Roman Krznaric thinks “sheer courage” can come in handy too, for instance in asking difficult questions that can improve your empathic accuracy.[31]

There is some tension between empathy and objectivity, impartiality, fairness, and justice. On the one hand, it can exacerbate biases that hurt such things; on the other hand it may help us more vividly see “both sides” of a case and may assist mercy and epeikeia.

The good and bad of empathy

There is extreme disagreement about the value of empathy: ranging from Karla McLaren, a self-described hyperempath, whose book calls empathy “Life’s Most Essential Skill,”[5] to Paul Bloom, whose Against Empathy says we’d be “better off without it.”[4] I’ll start with the case in favor of empathy.

What good is empathy?

“If there is one emotional intelligence skill that we would recommend developing, it’s definitely empathy. Empathetic people are happier, more self-aware, self-motivated, and optimistic. They cope better with stress, assert themselves when it is required, and are comfortable expressing their feelings.” ―Dr. Ilona Jerabek, president, PsychTests[32]

Empathy is (nearly) universal in humans. It develops in children in a predictable way at an early age. Forms or components of it are found in other animals. All of this gives the impression that it’s something that’s been selected for, and so is probably good for us at least in the for-our-reproductive-success sense of “good.” Why might this be? Here are a few common theories for how empathy might have been selected for:

  • Mirroring allows us to react more quickly to threats. When we see a fight-or-flight reaction in someone else, we can get our guard up before we know why.
  • Empathy helps us intuit and meet the survival needs of our ridiculously inept children.
  • Empathy also helps us as children to understand and manipulate the adults whom we rely on to get our needs met.
  • Empathy helps us navigate our social environment. It tells us whether someone means us help or harm, helps us better detect deception, helps us discern who is allied with whom, helps us distinguish an accidental jostle from a hostile poke, and so forth.

If you mimic the posture, facial expressions, and vocal style of someone you are with, you can thereby encourage them to help you and to form a favorable opinion of you.[33] Savvy business people consciously take advantage of this in job interviews, negotiations, and sales pitches. Police interrogators use this to build rapport with suspects. This may seem pretty far-removed from what we usually think of as empathy, but it may be more evidence for how the early mimicry/mirroring stages of empathy can help strengthen one’s social position.

Empathy seems to improve the quality of social relationships. People who score more highly on empathy questionnaires also report having more positive relationships with other people.[34] People also tend to value empathy in their friends and romantic partners. A friend goes up a notch in my book if it seems they “get where I’m coming from,” and I think demonstrating empathy is often an effective way for me to say that I care about a relationship and want to deepen it. Empathy might also improve your gift-giving proficiency.[35] 

Being empathic seems to correlate with feeling better about your life. In one longitudinal study of adults, researchers found that “[p]eople with higher empathy scores reported greater life satisfaction, more positive affect, less negative affect, and less depressive symptoms than people who had lower empathy scores.”[36]

Children who exhibit more empathy also have more resilience.[37] Empathically sampling other people’s situations and emotional states may help you to prepare for such things in your own life. This may also help you to learn culturally-legible ways to request an empathic response from others. When I roll my eyes and throw my hands up to heaven, I reenact a theatrical way of broadcasting “empathize with me in my exasperation!” that I had to learn somehow.

Thinking about other people’s problems can be a welcome distraction from your own (or can put your own into perspective), thus making you feel better. In one study, for example, peers who helped other peers with their depression found their own depression symptoms improving.[38] (Another explanation, but one that also favors empathy, is that when you give advice to others you may realize it’s good advice and take it yourself, but you wouldn’t have come up with it on your own for some reason.)

Empathy helps you to read the room, which can make you more courteous, persuasive, and so forth.

Empathically feeling unusual emotions in others (or in fictional characters) helps us to recognize them in ourselves and broadens our emotional intelligence. The more emotions we encounter in others, the more we are able to associate certain behavior and visible bodily changes with those emotions, and the better emotional vocabulary we develop. That helps us to potentially better understand ourselves, and also feeds back to improve our empathic accuracy.

When we look at the world through someone else’s eyes, we can use this as a mirror to look back at ourselves and get some idea of how we appear to others.

Empathy can be an aesthetic pleasure, “by widening the scope of that which we experience… by providing us with more than one perspective of a situation, thereby multiplying our experience… and… by intensifying that experience.”[39] It can sometimes be harmlessly pleasurable, for instance when we take joy at the joy of children discovering things or playing make-believe, or when we use empathy to satisfy our curiosity about other people’s lives.[40]

Paul Bloom suggests that empathy might work best in a backwards way: rather than a empathic person becoming distressed by seeing another person in distress, a person in distress may look at a bystander who is not so freaked out and become calmer as a result.[16] “Calm is contagious,” as the saying goes. For example, Leslie Jamison spent some time as a “medical actor” helping to teach med students to demonstrate empathy toward their patients. When the time came for her to be a genuine patient, she realized what she wanted from her doctor was different from that sort of empathy: “I wanted to look at him and see the opposite of my fear, not its echo.”[41]

Religious applications

Empathy is occasionally used to heighten religious feelings. Group worship and ritual (and chanting and song) can have the effect of synchronizing the worshipers’ emotions and outlook.

But there are also some practices that seem to use empathy to strengthen communion with the divine. For example, Christians may imagine themselves on the cross, in all of that gory awfulness, so as to better appreciate Christ’s sacrifice. Some Christians are said to have developed sympathetic wounds (stigmata) as the result of intense contemplation of the crucifixion. In some varieties of Buddhist tantric meditation, the meditator tries to merge their identity with a deity so completely as to lose their own identity.

Business applications

Empathy has become a business buzzword. It is supposed to have applications in management and marketing for example. Harvard Business Review (US) and The Empathy Business (UK) have produced “most empathetic” rankings of businesses. The “leadership consulting firm” DDI found in 2016 that 20% of U.S. employers offered empathy training to managers.[42]

If you can empathize with your (potential) customers, identify their problems, and come to understand how you can help them, you presumably can develop new products & services for the niches you thereby identify. This can be done through field observation & study of (potential) customers, and by empathically trying to get into their heads: why do they act the way they do; what are they trying to accomplish; how might we help them meet their goals more effectively? “It’s a process informed by deep qualitative data rather than statistical market data.”[43]

Patricia Moore was a pioneer of this technique. In one example, she used makeup and prosthetics to simulate the experience of elderly people, and used the insights she gained from this to inspire friendlier products for that customer segment.[44] Design engineers at Ford Motor Company wore prosthetics to simulate effects of pregnancy and of old age, in order to help them design cars that would work better for a broader set of customers.[45] Fidelity uses a virtual reality training application to put its phone bank “associates” in a (dramatized) customer’s home so they can see what it is like to be on the other side of their conversations.[46]

Customer empathy is exploited in advertising, which often invites us to empathize with characters in brief vignettes (who eventually buy product or service x and feel glad they have done so). See for example the old television ad below. It tells the story of a protagonist who meets and overcomes a challenge. We are meant to empathize with her in her struggle (and to imagine successfully seducing our husbands back with Yuban).

Charity marketing is particularly sensitive to the empathy⇒compassion⇒generosity pipeline. It is notorious for the sometimes ham-fisted ways it plays to the biases that accentuate human empathy, for example our propensity to empathize more with individuals than groups, or with the adorable over the homely:

We can expect marketing to become increasingly manipulative in this way, especially now that AI is being trained to intuit and respond to human emotions.[47] After all, even ELIZA was oddly engaging.

What about psychopaths?

A lack of empathy is a defining feature of the psychopath. Because of this, sometimes the psychopath is trotted out as an example of what empathy is good for (the implication being that lack of empathy is what made the psychopath the way they are).

But this is less clear-cut than it may seem. For one thing, the logical structure of that implication is faulty. (A lack of hair on the scalp is a defining feature of alopecia, but avoiding alopecia is not a good reason to let your hair grow long.)

For another, it isn’t entirely clear what the nature of the empathy deficit is in psychopaths. There is some evidence that psychopaths are perfectly capable of empathizing, but just don’t typically care to.[48] Psychopaths can be very manipulative and deceptive, which means they likely have good models of other people’s emotions, points-of-view, and so forth; they typically do fine on theory-of-mind tests.[49] They just don’t seem to give a damn about anyone else. On the other hand, as I was putting the finishing touches on this post, researchers announced that they found people with psychopathic tendencies are less likely to yawn contagiously, implying that the deficit may prefigure affective empathy.[50]

The abhorrent behavior we associate with psychopaths may have less to do with their supposed lack of empathy than with other things that diagnosed psychopaths typically have in common (such as poor impulse control, criminal history, and low emotional engagement).[4]

What bad is empathy?

“The problems we face as a society and as individuals are rarely due to lack of empathy. Actually, they are often due to too much of it.”

“If you are struggling with a moral decision and find yourself trying to feel someone else’s pain or pleasure, you should stop.” ―Paul Bloom[4]

Empathy boosters invited a backlash with some of their over-hyped claims. From reading Roman Krznaric’s Empathy: Why It Matters, and How to Get It, for example, you would think that without empathy, mothers would let their children starve, charities would dry up, and nobody would help anyone else or commiserate with suffering friends. Without empathy, we would be damned to a “heartless world of indifference”;[51] and yet even now we are plagued by an “empathy deficit” and “epidemic of narcissism” that explains everything from Syrian civil war to child-molesting priests to insufficient action on global warming.[52] “Empathy is like a universal solvent,” wrote researcher Simon Baron-Cohen. “Any problem immersed in empathy becomes soluble.”[53]

So now there are many strong and persuasive criticisms of empathy. From the point of view of empathy-as-a-virtue it is important to discern whether these describe ways empathy is wrong, or just ways empathy can go wrong. If the former, we should consider dropping empathy from the list of virtues we aspire to; if the latter, we need to carefully attend to how to be empathic more wisely and well.

Note also that some of these criticisms hinge on a particular definition of empathy. Paul Bloom, for example, in Against Empathy, mostly draws the line against affective empathy, and has kinder things to say about being understanding (part of cognitive empathy) and compassionate (part of behavioral empathy), which he thinks are valuable things that should be insulated from (affective) empathy in order that they should operate more effectively.

Some of the criticisms also push back against the idea that empathy is altruistic, or is the wellspring of altruism. They insist instead that empathy is really mostly for the benefit of the empath. From my point of view, though—examining empathy as a virtue—this is not damning in the least. I should hope that empathy helps the empath to thrive: then I know I’m not barking up the wrong tree.

All that said, here are many of the criticisms of empathy:

Empathy may make you more vulnerable to emotional predators and parasites. Dishonest panhandlers, for example, rather than asking for money outright will sometimes concoct an intricate “sob story” to encourage you to cough it up. If you are used to taking guidance from your empathy, you may be more likely to fall victim to such cons.

If you are not sufficiently critical in your empathic response, you may get caught up in someone else’s enthusiasm or rage when you would be wiser not to. This too can make you vulnerable to manipulation, both by individuals and by institutions that want you to help further their agendas. For example, it’s common for sympathetic victims of particular (real or imagined) atrocities to be repeatedly and vividly referenced in the rhetoric of those drumming up wars, pogroms, moral panics, and the like. When rational appeals to justice and interest fail to persuade, the unavenged suffering of that innocent child orphaned by the Hun might do the trick.

If you know you can evoke empathy in people, it can be difficult to avoid the temptation to play to the camera. Instead of feeling and behaving in an authentic manner, you feel and behave dramatically, with an eye to the effect this has on others. You see this even in young children, who may, for example, skin their knees and then look around to see if anyone else is watching before they decide whether or not to cry. (Is this a bad thing, or is it just a human thing? Is there such a thing as an authentic, genuine emotion that precedes its expression, or are emotions essentially signaling devices from the get-go?)

Empathy can discourage you from making necessary decisions that hurt or disappoint other people (for example, the decision to subject your child to a vaccination, or to say “no” when asked for an imposing favor).

Empathy may enforce conformity with other people’s standards, and discourage independent thought. If you would empathically feel someone else’s disappointment, disgust, judgment, or upset, in an unpleasant way, this may encourage you to conform to their desires and expectations instead, even when non-conformity would have been a better option.

The aesthetic appeal of empathy, or the emotional fix you can get from it, can be intoxicating and can sometimes cause what looks like pathological empathy-seeking behavior. This may be a factor behind things like outrage porn, poverty porn, and the popularity of videos capturing other people’s embarrassments, failures, and Jackass-style injury-inflictions. On the other hand, this same appeal probably explains a lot of the attraction of literature and drama. For example: when you read a mystery story you want to share the curious befuddlement and eventual a-ha of the intrepid detective. That’s part of the fun.

Empathy can be a mirage. People seem prone to believe their empathic intuitions are more accurate than they really are (see “Empathic accuracy” above). If you act on the strength of such belief without adjusting for this bias, you may act poorly. If you have an exaggerated belief in your empathic powers, you may lose the curiosity you need to really know what’s going on with someone else.

Clumsily-expressed or overconfident empathy can seem presumptuous and condescending. A politician turns on the puppy-dog eyes and tells me “I feel your pain.” Do you really? Leslie Jamison, who acted the part of a patient to help physicians in training with their bedside manner, rankled at the awkward tropes of empathy they were trying out:

I grow accustomed to comments that feel aggressive in their formulaic insistence: that must really be hard [to have a dying baby], that must really be hard [to be afraid you’ll have another seizure in the middle of the grocery store], that must really be hard [to carry in your uterus the bacterial evidence of cheating on your husband]. … “I am sorry to hear that you are experiencing an excruciating pain in your abdomen,” one says. “It must be uncomfortable.” [54]

Empathy can be emotionally wearying, especially in the case of distressing emotions or situations. Medical caregivers for example may need to suppress empathy in order to avoid burnout.[55] It seems people only have so much empathy to give, and so they need to ration it. If you use up your empathy at work (or in fretting over the benighted people of Borrioboola-Ghâ), you might not have any left for your family.[56]

If you cannot control your empathy, you may instead turn away from things that could really use your attention, if those things trigger an unpleasant empathic response you want to avoid. And that can encourage you to come up with quasi-rational excuses for not caring about those things, which can make you irrationally callous or neglectful. A less empathic, more level-headed problem-solving approach might have more staying power and therefore be more helpful.

To the Stoic philosophers, to condition your emotional disposition on the emotions or fortunes of someone else would seem to be foolishness of the first order. Cicero, summarizing (though not fully endorsing) the Stoic point of view, said that someone who feels distress at another’s misfortune is committing as much of an error as an envious person who feels distress at another’s good fortune.[57]

Is empathy unnecessary?

Empathy (feeling-with) is often defended as important in the development of compassion (feeling-for). Paul Bloom wonders why we cannot simply be compassionate without jumping through the biased and disorienting hoops of empathy first? When we are happy about our children’s triumphs, why feel the need to first be triumphant when we can just be whole-heartedly glad for them? When someone close to us is sad or nervous, what’s wrong with going straight to being consolatory or calming rather than mirroring their sadness or nervousness back at them first?[16]

Bloom shares the results of an fMRI study of a Buddhist meditator (Matthieu Ricard) doing “various types of compassion meditation directed toward people who are suffering” and reports that this “did not activate those parts of the brain associated with empathic distress—those that are normally activated by non-meditators when they think about others’ pain.” The same meditator “put himself in an empathic state” and got the more typical results, but also found it comparatively exhausting. Experiments on meditation-naive people in which they’re trained in either compassionate or empathic mediation apparently show something similar: the former is more pleasant and also leads to kinder behavior.[4]

Some researchers have had difficulty finding the expected evidence that empathy makes you a better person, or that lack of empathy makes you worse.[58]

Empathy does seem to be effective at goading people to altruism, but its biases and flaws mean that this altruism is often poorly-targeted and -executed. The cases of Baby Jessica who was rescued after falling down a well (“sympathetic strangers showered the family with teddy bears, homemade gifts, cards, and cash”[59]) and the #Kony2012 phenomenon are examples of empathy-induced altruism being both impressively strong and questionably-targeted.

Empathy is biased

Empathy is demonstrably biased in terms of whom we are likely to empathize with and in what situations and in what manner. These biases are hard to defend as bases for our compassion, kindness, respect, and so forth. If we rely on empathy to guide our decisions and priorities, we may unthinkingly import those biases. These are some of the biases that have been documented:

Empathy causes unconscious favoritism

If you feel empathy towards someone, this may cause you to practice unjust favoritism towards that person without recognizing that you are doing so.[60]

Empathy seems to encourage us to take sides with whomever we empathize with first. Fritz Breithaupt goes so far as to claim that “we do not act morally because we feel empathy; rather, we moralize to justify our quick and empathetic side-taking.”[3]

This bias is subject to exploitation. When you hear a party to a conflict say something along the lines of “you can’t stand aside at a time like this; you have to take a side!” you’re also being implicitly told “and try on my side for size first” which then stacks the deck. Once someone decides to empathize (with e.g. Brett Kavanaugh or Christine Blasey Ford), that person is likely to view emerging evidence from an empathically-biased perspective and to be less able to evaluate it on its merits. In this way, “empathy not only fails to end conflicts, but deepens them.”[20]

Another way this plays out is that people who score higher on empathy scales are more likely to advocate harsher punishments of those who transgressed against whomever they are empathizing with.[61]

Your empathy is likely biased against people who are different or who are in the outgroup

People empathize more with those who are like them in certain ways (such as the usual suspects of language, culture, race, and nationality).

This effect is even measurable at early, pre-cognitive stages of empathy. For example, it’s easier to recognize emotions in ingroup members than in outgroup members (even when the ingroup/outgroup distinction is artificially imposed, not based on previous experience).[62]

The “othering” that applies to empathy (and other things) is somewhat flexible. “For example, in research where temporary group identifications were arbitrarily manufactured, dominant group identity such as race became secondary.”[63]

A good case can be made that this bias contributes to loyalty and group cohesion / coordination (and perhaps this is why we are biased in this way). In a conflict, you empathize most readily with those who are like you or in your ingroup, that triggers empathy’s side-taking bias, and that causes you to line up with your squad in the conflict.

We are also less likely to empathize with people we envy or otherwise dislike.[64]

Since empathy has biases that make you more likely to empathize with someone who is like you or for whom you have fonder feelings, when you empathize with someone you may thereby inadvertently suggest to others that you feel yourself to be like them or are fond of them. This can make it costly to (and can disincentivize you to) empathize with unpopular people. Why are you empathizing with that person convicted of possessing kiddie-porn? You some kind of pervert-lover?

The internet allows us to discover people who are uncannily like us in very specific ways. I wonder if this raised the bar for whom we see as similar-enough to empathize with? Since most people we meet aren’t like the select people we’ve become companions with on-line, are we now more apt to find we “can’t relate” to them?

Your empathy is subject to change with your social/political power

People tend to empathize less when they have more social or political power. Indeed lack of empathy may be a kind of status symbol.

Melania Trump visited a detention center for migrant children while wearing a jacket painted with the words “I REALLY DON’T CARE, DO U?”

This effect is most noticeable in extreme cases, such as when an abusive parent or a hostage-taker has arbitrary power over another person. The Stockholm Syndrome is one way this can play out. The vulnerability of the hostages drives them to extremes of empathic awareness so they can try to anticipate their captors’ actions. This has such a strong effect that the captives begin to sympathize with and defend their captors.

But this is also measurable in less-extreme circumstances. For example, people from lower-strata economic backgrounds exhibit better empathic accuracy (judging others’ emotions) than those from higher-strata economic backgrounds.[65]

We should be cautious, given how some “priming”-style social science studies failed to replicate, but with that disclaimer attached: In a variety of experiments, people who were asked to recall a situation in which they had power over someone else then demonstrated reduced ability to mirror others, to comprehend their viewpoints, or to learn from others’ perspectives.[66]

However for some people who attain power, “if they feel responsible for those who have less power or they already value being empathetic” they can be more than typically empathic, counter to the usual trend.[67]

Given the various flaws and biases associated with empathy, maybe it’s good that powerful people typically don’t empathize as much or as well. Maybe this helps them make tough, rip-the-bandaid-off decisions that cause short-term pain for long-term gain. Or maybe their lack of empathy helped them make more rational decisions that also helped to empower them.

There are many suggestions for why people exhibit this bias, among which are these:

  • If you’re already high on the ladder, you don’t need to put as much effort into understanding others. You can relax; you’ve made it.
  • If you have power, you have authority and coercion on your side and you don’t need persuasion as much, so you don’t need to get inside others’ heads to get them to do what you want.
  • It’s a valuable survival skill to understand the inner workings of those in the dominant culture if you’re not in it; it’s not so valuable to understand the dominated culture if you’re in the dominant one.
  • This could be an effect of a pyramid-shaped hierarchy with more powerless people and fewer powerful ones. It’s easier for each employee to be concerned, say, about whether their manager woke up on the wrong side of the bed, than it is for the manager to keep track of all of the various headspaces of their many employees. (However, supervisors seem to score higher on accurately recognizing the emotions of others than their reports do.)[68]
  • Some leaders have a management style that involves giving direction in terms of vaguely-stated aspirational goals, and then humiliating and browbeating subordinates who disappoint them. The subordinates are thereby incentivized to “get out ahead” of what the leader explicitly asks for and to try to anticipate what will make the leader happy instead. This can require those subordinates to devote a lot of extra attention to empathic mind-reading.

Other empathy biases

People prefer to empathize in “clear, relevant, and decisive” situations.[3] And we do so in a way that makes them clearer, more relevant, and more decisive—that is, in a way that papers-over ambiguities and nuances that don’t fit a simple story.

We tend to empathize with the here-and-now, with what is immediately available to our senses. Empathy is good at prompting prosocial behaviors that are informal, unplanned, and directed at someone right here right now, but not very good at prompting things that require more abstract, long-term concern (like giving blood, donating to charity, or volunteering).[69] We also have a harder time empathizing with what is not immediately apparent (for example, with the struggles of someone with an outwardly invisible brain injury). Empathy can operate on the “out of sight, out of mind” principle.

Empathy operates best at the level of the individual, and so we may empathize with one person facing a plight, but if we hear about multiple people undergoing a similar plight, empathy has a harder time gaining a foothold. (Mother Theresa: “If I look at the mass, I will never act. If I look at the one, I will.”)

Empathy favors known/nameable/picturable individuals over unknown/anonymous/invisible ones. An example of how this can pervert decision-making is vaccine reluctance. If some child has a bad reaction to a vaccine, that child is known, has a face and a name, and can therefore be more-easily empathized with. But if a child avoids catching a deadly or crippling disease thanks to vaccines, well, can you even point to them? They’re a statistical projection, barely even a rumor so far as empathy is concerned.

People with different levels of experience empathize differently: A child may not empathize with a situation because they do not yet know what its relevant features are and have not had similar experiences of their own to compare it to, while a world-weary person may find it hard to empathize with the anxiety and surprise of a tyro.

We empathize more with people in distress whom we feel are “innocent”—that is, they were not responsible for bringing the distress upon themselves, or in any case that distress is disproportionate to any cause they participated in.[70]

We are less likely to empathize in “hopeless cases” of distress, in which we don’t think anything can be done to make it better.

It may not be realistic to say “be empathetic, but try to be on guard about your biases.” Paul Bloom thinks advice like that is hogwash, especially when people propose deploying empathy to mitigate the harmful effects of ingroup/outgroup biases. “Asking people to feel as much empathy for an enemy as for their own child is like asking them to feel as much hunger for a dog turd as for an apple—it’s logically possible, but it doesn’t reflect the normal functioning of the human mind.”[71]

Vampiristic and sadistic empathy

Fritz Breithaupt highlights two harmful forms of empathy.

One, he calls “vampiristic empathy.”[72] This sort of empathy is more aggressive in how it attempts to import the experience of another person. The empathic vampire is not content with sampling that experience in order to understand it better, but they want to go further and appropriate that experience for their own. Examples of this include “helicopter parents, stage mothers, …fans…[, and] stalkers” who engage in “obsessive observation” of their subjects “while supplanting [the other’s] objectives, goals, or desires with [the empathizer]’s own.” The goal of this sort of empathy is to enrich the self with something that is envied about the other. For example, obsessive celebrity fans empathize with the objects of their obsessions in the hopes of feeling what it’s like to be special, worthy of admiration, and worthy in particular of the obsessiveness of the fan. But this technique is self-frustrating. By obsessively concentrating on the other, the self just gets emptier, and the quest to assimilate the other person’s life becomes increasingly desperate.

The other he calls “empathetic sadism.”[73] Some people get a charge out of empathically feeling the suffering of someone else. And many of us can be tempted by, for example, wanting to see someone who has hurt us suffer as we suffered. Breithaupt wants us to recognize that in such cases empathy is what allows us to imagine what will most hurt our victim, and also allows us to revel in that pain.

In empathetic sadism “[a] person creates, encourages, wishes for, or tolerates a scenario in which someone else is placed in danger or made to suffer, precisely in order to feel empathy with that person, now cast in the role of the victim.” And this is not limited to brutal psychopaths or revenge fantasies. Why do we enjoy daredevils, like for example tightrope walkers or Evel Knievel-style stunt performers? We enjoy safely empathizing with their peril, maybe with their courage or fear—which they have taken on for our entertainment. Is this not a sadistic form of empathy?

Breithaupt himself puts forward “empathy rape” as an example. Contrary to theories that the rapist treats their victim as an object, devoid of feelings, Breithaupt says that at least to some rapists, the opportunity to empathize with the suffering of the victim is part of the motivation for the crime. He points out that “non-consensual” porn fantasy stories attend in detail to the feelings of the victim (a typical trope is for the victim to start off horrified and in pain, and end up begrudgingly delighted and humiliatingly grateful). Though I suspect that non-con porn fantasy stories probably represent genuine rape at least as unrealistically as other porn fantasies represent their real-world analogues.

There are also milder forms of “manipulative empathy”—“behaviors on the part of the empathizer intended to guide the other into a particular situation in which they will be emotionally predictable and it will be possible for the empathizer to coexperience their emotions.” Consider for example jumping out from behind a bush to scare someone, or telling a puzzling riddle. Internet trolls who try to push people’s buttons on-line are another example.

Sometimes people cooperate to converge on a shared and predictable emotional state: for example when you sing in a choir, attend a concert, watch a thriller, or attend a football game. There is something valued about the shared emotional experience (it wouldn’t be nearly so fun to go to a rock concert or a football game if you were the only one in the audience, even if that meant you had a front row seat and never had to wait in a long line to get a beer).

What’s common to all of these is “empathy for empathy’s sake. The empathetic response becomes its own goal, independent of any consideration for—or, indeed, detrimental to—the well-being of the other.”

A safer outlet for the sadistic empathy impulse is (theatrical) tragedy. Why do we enjoy a tragedy like Hamlet, in which everyone we care about over the course of the play dies and everything goes to shit? Our empathy for the characters evidently does not mean we are disappointed at their unhappy endings. This is a clue that it’s “empathy for empathy’s sake” we’re after. “[F]iction allows us to enjoy empathy without compassion or obligation to help.”

The Nietzschean critique

Fritz Breithaupt also analyzed empathy from a Nietzschean perspective.[74] This is my summary of his summary of Nietzsche’s ideas, so salt to taste:

In Nietzsche’s view, people tend to have poorly-developed, weak senses of self, to their detriment. When you empathize with someone, you further suppress your self in order to simulate their point of view (other ways in which you might respond that do not weaken the self would include loving or hating that person, praising or condemning that person, helping or hindering that person, or judging that person from your own point of view).

Empathy simplifies the other person to make them more comprehensible, and this process of simplification exaggerates how unified and coherent the other person is: in other words, it makes their self look stronger than it really is. The empathic person “project[s] onto others the self that they are lacking” and “feels empathy for that which they must give up in order to be able to feel empathy: a strong self.” This makes empathy somewhat vampiric in the sense described above, and also makes it a source of envy: we can dislike those we empathize with because we envy the strong self we believe they have.

Breithaupt calls this the “empathetic endowment effect” and suggests that it explains, among other things, why people like to follow the goings-on of celebrity stars and charismatic politicians. I wonder if this also explains some of the “why we love sociopaths” phenomenon. Sociopaths, because they typically don’t display much empathy, don’t dissolve their selves in other people the way most of us do. This may suggest to us that they have the strong selves that we lack and envy and fantasize about having.

Nietzsche sees people’s attempts to be “objective” as a variety of this same disorder. To see something objectively is sort of like empathizing with the point of view of God, and it is subject to the same sort of distortions as empathizing with another person (for example, we can be tempted to project a strong self onto this God that we hypothesize). If you try to be “objective” you undermine your self: You see your own priorities, passions, and judgments as just one of many, not as of any particular importance. You distrust and denigrate what is merely subjectively yours. You don’t form firm opinions because after all there are diverse opinions to take into account, and objectively no values by which to distinguish good and bad. (It sounds to me like Nietzsche would find the attempt to counter biased empathic engagement with rational “effective altruism” just makes the problem worse.)

How can you improve at empathy?

Since empathy seems to be a composite skill, we should be on the lookout both for things that may improve empathy writ large and for things that may improve the functioning or interoperation of its component parts. A good first step for improving your empathy might be to examine more closely which components of your empathy need the most improvement.

There is plenty of advice on how to become better at empathy and its components (and, less usefully, on how to become simply more empathic). And there is evidence that empathy is a skill that can improve with deliberate training.[75] I’ll try to summarize some of this advice here.

Asking questions, and listening well

As touched on in the “empathic accuracy” section above, one way to improve your empathy is to use the gift of language. You don’t have to painstakingly intuit what another person is thinking and feeling if they’re happy to just tell you when prompted.

This is a two-part skill on the empath’s part: asking the right questions, and attending well to the answers. It also only works well with a cooperative partner who is not being reticent or deceptive (or poorly self-aware or emotionally inarticulate).

“Active Listening” and “Nonviolent Communication” techniques are often recommended for this purpose. Some of this stuff can seem off-putting if it’s not used with a gentle touch: “they’re very obvious techniques, and I could see through them even as a toddler,” says Karla McLaren “You don’t have to parrot me to empathize with me. You need to interact—honestly, authentically, and as yourself.”[76]

Sometimes people engage in small talk with the goal of avoiding conversation that will be emotionally taxing. When we want to empathize, we need to shift from that mode to asking questions that go more to the heart of things and that invite more revealing answers. “Empathy isn’t just remembering to say that must really be hard,” says Leslie Jamison. “It’s figuring out how to bring difficulty into the light so it can be seen at all. Empathy isn’t just listening, it’s asking questions whose answers need to be listened to.”[28]

Another thing that can help is a broad emotional vocabulary that, for instance, doesn’t have to fall back on something vague like “excited” when you really mean something more precise like frantic, aroused, expectant, agitated, enthused, or manic. Karla McLaren insists that “empathy is first and foremost an emotional skill.”[77] She, and other writers on the subject, recommend improving your emotional intelligence skills as a way of also improving your empathy.

If you want to practice asking good questions and listening well, you can borrow a human from the Human Library and do just that. Or you might go out for a Conversation Meal. If that seems like too much fuss or involves too much vulnerability, there’s always Chatroulette, which you can enjoy over your computer from the comfort of home.

Karla McLaren also recommends the back-channel communication technique of gossip. Gossip can give you important context about the people around you that you might not be able to learn directly from them and that can help you improve your empathic accuracy about them. Gossip has a bad reputation, for instance because it can surface things like envy and jealousy and because people dislike feeling gossiped about, and malicious gossip can be pretty awful, but good gossip can be informative and helpful in the empathic project.[78]

Literature, film, and other such media

“The greatest benefits we owe to the artist, whether painter, poet, or novelist, is the extension of our sympathies. Appeals founded on generalizations and statistics require a sympathy ready-made, a moral sentiment already in activity; but a picture of human life such as a great artist can give, surprises even the trivial and the selfish into that attention to what is apart from themselves, which may be called the raw material of moral sentiment.… Art is the nearest thing to life; it is a mode of amplifying experience and extending our contact with our fellow-men beyond the bounds of our personal lot.” ―George Eliot[79]

“It’s not a big leap to suppose that the habit of reading other people’s words could put one in the habit of entering other people’s minds, including their pleasures and pains.” ―Steven Pinker[80]

When you engage with good literature (or film, or what-have-you), you will often empathize with its characters. Consider a movie in which the protagonist is at a crossroads; there are pros and cons to both decisions, and the stakes are serious. The protagonist pauses, anxious and indecisive. If it’s a mundane but okay movie you’ll be curious about what happens next. If it’s a great movie, you’ll be anxious and indecisive along with the protagonist.

Good literature can also let you inside of other people’s heads in a way that is normally not possible in real life. Consider stream-of-consciousness writing like Virginia Woolf’s Mrs. Dalloway or To the Lighthouse. Woolf shares her extraordinary talent for empathy with us—takes us by the hand and lets us empathize vividly too, by illuminating the insides of her characters for us.

Literature can help you learn what it feels like to empathize with someone very different from yourself or in a very different situation than yourself, when you otherwise might not have known where to start. The author can also spell out for you certain social nuances that you might otherwise have difficulty picking up on your own, and thus prime you to be aware of them in similar real-life situations in the future. Some literature operates as a kind of mystery story in which you have to be attentive to clues that help you piece together the motivations and attitudes of various characters; such fiction may prepare you for real-world challenges of empathic accuracy. In short, maybe literature is good practice for empathy.

A meta-analysis designed to determine whether existing research supports the claim that fiction-reading causally improves social cognition determined that “fiction reading leads to a small, statistically significant improvement in social cognitive performance.”[81]

Lynn Hunt argued in Inventing Human Rights: A History that the concept of human rights developed how it did and when it did in part as a result of the influence of mid-eighteenth-century European novelists, particularly those whose use of the briefly-in-vogue “epistolatory novel” form gave readers a more vivid sense that they were gaining access to the candid details of a real life. These novels became something of a craze, and the culture was swept up in this new, shared experience of empathizing with a (fictional) person in unaccustomed intimacy.[82] “The epistolatory novel did not just reflect important cultural and social changes of the time. Novel reading actually helped create new kinds of feelings including a recognition of shared psychological experiences, and these feelings then translated into new cultural and social movements including human rights.”[83]

This sort of thing may be true of non-fiction as well. There’s a genre of non-fiction (sometimes called “role reporting”) that describes the empathic process by having the authors immerse themselves in an unfamiliar lifestyle in order to see it from within with fresh eyes. I’m thinking of things like John Howard Griffin’s Black Like Me, George Orwell’s Down and Out in Paris and London, Barbara Ehrenreich’s Nickel and Dimed, and Günter Wallraff’s Lowest of the Low. This is a sort of empathy-extreme. It is designed explicitly to help the reader empathize with the people the author has learned to empathize with, but may also model the process of empathizing more generally. (An interesting fictional example of where extreme empathy is central to the plot, rather than incidental to the storytelling, is the film Being John Malkovich.) Biography and autobiography also seem designed to satisfy that empathic urge in us to know what was it like to [grow up as a reincarnated lama, walk on the moon, cross the continent in a covered wagon, etc.]. 

One problem with this sort of empathy is that it may leave you at the mercy of cultural creators, who may not be very careful in how they form and direct your empathy. Your child’s developing empathy may be piqued by reading The Diary of Anne Frank or by The Turner Diaries, depending on which one gets into their hands first. Only certain sorts of stories and certain sorts of protagonists can get Hollywood budgets.

Sometimes works of literature use shortcuts to help you quickly decide whom to empathize with and whom you can disregard as an NPC or obstacle (the villain has a scar, speaks with an accent, and so forth), and these may influence your real-world empathy in ways you wouldn’t want. Actors exaggerate their body language & other emotional expressions, in order to make them easier for us to read. There may be some danger that this teaches you only an artificial stage-dialect of emotions that translates awkwardly to the real world (imagine trying to understand people if all you knew about them was what you learned from Noh drama). However I suspect people who have grown up in the post-television era have learned how to express their emotions in part by what actors have modeled for them, so there’s some convergence between the artificial and the real; maybe this just helped people adopt a common dialect.

Fiction writers sometimes try to get their readers/viewers to empathize with a morally repulsive character. (And here I’ll plug Why We Love Sociopaths again.) That’s all fun and games if fiction is just recreation, but if it does indeed help to shape our empathy, do we really want our empathy shaped in such a way that we see-things-from-their-side when they are a brutal serial killer or what have you?

Empathy may be a limited resource. If you do too much of it, you have to recharge before you can do it again. For this reason Karla McLaren is concerned that people turn to potentially empathy-draining activities like fiction, movies, television, video games, or outrage porn to relax. She recommends that you cultivate some non-emotionally-receptive, solitary recreation options—and that you create physical spaces where you can retreat and escape from the intrusion of others’ emotions—if you find your empathy waning.[84]

Because fiction allows us to empathize with a character in a way that keeps us actually aloof from them, it could also conceivably atrophy behavioral empathy. When you empathize with a fictional character, that never requires anything further from you. (In this way, it is a form of the narcissistic “empathy for empathy’s sake” that Fritz Breithaupt identifies as “dark empathy”.)

Walk a mile in their moccasins

The aforementioned “role reporting” authors went to great lengths to experience the lives they were attempting to empathize with. Sometimes people will try to boost their empathy by doing more limited, short-term performances of that sort. For example, they might spend one night sleeping rough on the streets in an attempt to gain more empathy for homeless people, or live on the spending allowed by a minimum-wage income for a week, or something of that sort. The Dialogue in the Dark museum exhibit takes place in total darkness, with a blind museum guide. Such things have a stunty feel to them and can be easily mocked, but there’s something to the idea that you can understand more about another person’s life if you directly experience some representative elements of that life.

However, this may not reliably produce the effects you might expect (empathy that leads to compassion). Counterintuitively, one study “found that people who have endured challenges in the past… were less likely to show compassion for someone facing the same struggle, compared with people with no experience in that particular situation” and that “people who have endured a difficult experience are particularly likely to penalize those who struggle to cope with a similar ordeal.”[85] Why? One explanation is that the rosy retrospection bias makes people forget how hard past difficulties were.

If you have more experience of life, you may “know how that sort of thing always turns out” and find it difficult to empathize with someone going through it for the first time.[3] Your adolescent child may indeed feel that their latest crush or craze is the most imperative thing in the world, but try as you empathically may, and even having experienced your own momentous adolescent enthusiasms in your time, you may just be unable to take it seriously enough to go there with them.

Acting (for instance, improv) and role-playing games might also help you to practice empathy by slipping into and embodying characters who live lives, and have outlooks different from your own.

Imagine ways in which the other is not so other

One possible way to overcome the ingroup/outgroup or othering biases that make it difficult to empathize with someone different or distant from yourself is to try to imagine them as though they were not so different or distant. Children are sometimes helped to develop empathy through what is called “multiple empathizing”: they are encouraged to imagine that some stranger in distress were really (say) their own mother, and how they would then feel.

In Matthew 25:31–46, Christians are told that Jesus will judge them in the last days in this way:

Then the King will say to those at His right hand, “Come, you blessed of My Father, inherit the kingdom prepared for you since the foundation of the world. For I was hungry and you gave Me food, I was thirsty and you gave Me drink, I was a stranger and you took Me in. I was naked and you clothed Me, I was sick and you visited Me, I was in prison and you came to Me.”

Then the righteous will answer Him, “Lord, when did we see You hungry and feed You, or thirsty and give You drink? When did we see You a stranger and take You in, or naked and clothe You? And when did we see You sick or in prison and come to You?”

The King will answer, “Truly I say to you, as you have done it for one of the least of these brothers of Mine, you have done it for Me.”

So Christians will sometimes strengthen their empathy by “Jesusing” the other. Mother Theresa again: “I see Jesus in every human being. I say to myself, this is hungry Jesus, I must feed him. This is sick Jesus. This one has leprosy or gangrene; I must wash him and tend to him.”

If you find yourself complaining “I just can’t seem to empathize with such-and-such people,” try instead to isolate a single such-and-such person to empathize with. It’s much easier to empathize with a particular person than with a group of people.

Then, consider what you share with that person, even at the most basic level. This is one of the most common and ancient empathy-related interventions. Consider Shylock’s plea in Shakespeare’s Merchant of Venice:

“Hath not a Jew eyes? Hath not a Jew hands, organs, dimensions, senses, affections, passions? Fed with the same food, hurt with the same weapons, subject to the same diseases, healed by the same means, warmed and cooled by the same winter and summer, as a Christian is? If you prick us, do we not bleed? If you tickle us, do we not laugh? If you poison us, do we not die?”

“The shift to seeing others not as different but as similar seems to be the strongest way to influence empathic resonance or insight.”[86] There’s also a potential feedback effect, in that the more you empathize with someone the more you recognize (or suppose that you recognize) similarities between them and you.

If you somehow come to feel empathy toward a member of an outgroup, this can also increase your propensity to feel empathy toward other members of the outgroup.[87] And if you learn examples of other members of your ingroup helping members of the outgroup, this can also increase your empathy for members of the outgroup.[88]

Try to see things from both/all sides

As mentioned before, when you empathize with someone, you tend to take their side, and this can make it difficult to evaluate conflicts dispassionately. One possible remedy is to make an effort to see things from other sides as well. Ideological Turing Tests are one way of testing your empathy in this regard.

But it’s not clear that this sort of intervention is reliable. In one experiment, grade school students in Northern Ireland were taught to understand the historical conflict there from the perspectives of Catholics and of Protestants, in such a way that the students were able to skillfully explain both perspectives. But this did not seem to reduce polarization in the students. Instead it seemed to help the students in “assimilating the experience of the other into their own frame narrative.”[89] It’s even possible that asking the students to employ empathy to process additional examples of the conflict (though seen from various points of view) exacerbated their original empathy-induced side-taking bias.

Be cautious around demands for empathy

When someone pleads with you to show empathy, they can make it sound like they are asking nothing more from you than what you owe them: to see things from their point of view, to show some consideration for how they feel.

But empathy can be taxing. It is emotionally costly. It is not just anyone’s for the taking. You have the right to ration it out as your wisdom and priorities dictate.

As mentioned above, the “see things from my point of view” gambit can be a way to encourage side-taking. There are times when you do not want to begin by taking sides but instead you want to evaluate a conflict according to impartial standards of justice. In such cases it may be best to say, “no; I’m going to keep looking from my own point of view.”

Some people who are having a bad time think they will feel better if they can make other people around them feel miserable too. Thus the adage “misery loves company.” Pleas for empathy can be part of that game. Sometimes people will crave the attention and intimacy they receive from others who empathize with them in their distress, and this incentivizes them to undergo (or masquerade, or amplify) additional distress. If you are cautious around demands for empathy, you may avoid feeding dumpster fires like these.

Games

Multi-person games sometimes incentivize trying to “get into the head” of your opponent or teammate. Even rock-paper-scissors has something of that. Bidding strategy in Bridge is a sort of formalized gameplay-empathy in which you use gameplay methods to intuit what your partner knows. There’s a frequently told tennis legend in which Andre Agassi was able to take advantage of a “tell” in Boris Becker’s body language to determine how Becker was going to serve.[90]

There are also a variety of games that are meant to be icebreakers or intimacy-builders. Sometimes these can provoke empathic responses, and some are even designed with this in mind. Questions & Empathy, for example, is a card game that is designed to get people to share less superficial parts of themselves by answering probing questions from unfamiliar perspectives. “It’s like a highbrow Cards Against Humanity. It escalates you from small talk to big talk ultra fast.”[91]

Prosthetics, virtual reality, and video games

Prosthetics can allow you to viscerally experience some aspects of lives that are different from your own. There are, for example, fake bellies that people can wear to empathize with pregnant people. A device called Sympulse transmits tremors from a Parkinson’s patient to another person, “to help foster clinical empathy… to give movement disorder physicians and caregivers a sense of what their patient or loved one is experiencing in real time.”

Invisible disabilities are particularly hard to empathize with. Detour: Brain Deconstruction Area Ahead is a (1994) film by someone who tried to show from their perspective how their traumatic brain injury causes distortions in how they perceive the world and the difficulties this causes.

Kaitlin Ugolik Phillips’s The Future of Feeling: Building Empathy in a Tech-Obsessed World describes a number of virtual reality simulations designed to prompt empathy. (Some of these are virtual reality in the sense of fully-immersive headset-style presentations, others are more like interactive 360° video essays. Some are dramatizations, while others are more journalistic attempts to place you at a real scene.)

  • 1000 Cut Journey — “participants embody a Black male, Michael Sterling, experiencing racism as a child through disciplinary action in the classroom, as an adolescent encountering the police, and as a young adult experiencing workplace discrimination”
  • 6×9 — “places you inside a US solitary confinement prison cell and tells the story of the psychological damage that can ensue from isolation”
  • Across the Line — “put[s] viewers in the shoes of a patient entering a health center for a safe and legal abortion”
  • After Solitary — “lets viewers walk around the cell with Kenny as he recounts his experiences in solitary”
  • The Alfred Lab — “embody Alfred, a 74-year-old African American man with macular degeneration and high-frequency hearing loss as he spends time with family, visits the doctor, and receives a diagnosis”
  • Becoming Homeless: A Human Experience — “spend days in the life of someone who can no longer afford a home”
  • Carne y Arena — “walk in a vast space and thoroughly live a fragment of the refugees’s personal journeys”
  • Clouds Over Sidra — “Meet Sidra. This charming 12-year-old girl will guide you through her temporary home: The Zaatari Refugee Camp in Jordan.”
  • The Displaced — “portrays the impact of war and displacement on children with heartbreaking, immersive realism”
  • Enter the Room — “experience the trauma of war through augmented reality”
  • Hunger in Los Angeles — “simulate the experience of watching a man go into diabetic shock at a Los Angeles food bank”
  • Project Syria — “witness a bomb go off in the streets of Syria, turning a normal Syrian afternoon to complete chaos and destruction”
  • Use of Force — “the homicide of Anastacio Hernandez Rojas who was beaten and tasered by more than a dozen border patrol agents”

Another source of interesting experiments in empathy-evoking technology is empathymuseum.com.

I find some of this uncomfortably aggressive. I imagine someone saying “you ought to empathize with so-and-so, and we’re going to keep giving you the helmet until you do.” Advocacy documentaries can be manipulative enough without adding VR to the mix.

I also wonder, why on earth would anyone volunteer to “experience the trauma of war”? What kind of monster would design an app whose purpose is to inflict the experience of the trauma of war on people? (I suppose since there are whole industries dedicated to inflicting the real trauma of war on people, I shouldn’t be so shocked.) Either the viewers know that they’re really only going to get a voyeur’s peek at it (which kind of defeats the purpose) or they imagine they’re going to get the real thing but they naively think of the real thing as though it were something safely scary like a roller-coaster (and is VR likely to change their mind about that?).

Something like Across the Line does not seem to me as though it is meant to help people learn to empathize with a patient getting an abortion. It strikes me more as a sort of waving the bloody shirt tactic to further raise the ire of pro-choice activists toward their opponents. As one critic said of Carne y Arena, it “puts so many eggs in the basket of creating empathy, since its power so clearly depends on a foundation of preexisting sympathy.”[92]

Video games are a sort of virtual-reality-lite. In “first person”-style games in particular it’s difficult not to feel-with the struggles and perils of the character whose actions you control and whose eyes you see through. Does this exercise the same skills as empathy? (Does it do so to our benefit?)

Developers have also designed video games that are meant to teach or improve empathy. Several examples I’ve seen seem to be designed for children and adolescents.[93] Another, Life is Strange, gave the game’s protagonist “an ability to read, experience, and manipulate the emotions of her peers” as part of the gameplay dynamics.[94]

Dealing with hyperempathy

Karla McLaren says she was hyperempathic as a child and that this caused some difficulties for her (for instance she would sometimes shut down or behave strangely in social situations because of this exceptional sensitivity). While she has now turned this to her advantage by becoming particularly skilled in and understanding of empathy, it was a painstaking process to get to that point. Her book The Art of Empathy: A Complete Guide to Life’s Most Essential Skill might prove inspiring to you if you have similar struggles.

I am far from hyperempathic, but I think I can empathize somewhat with this. I can remember times during my days of psychedelics enthusiasm when, under the influence of some drug or other, I became hyperaware of social nuances, subtext, and the multiple levels on which social interactions were simultaneously operating. All of the wheels-within-wheels of conversation and body language and insinuation and so forth left me reeling, unable to make a move in this 10-D chess game because I couldn’t possibly trace all of the ramifications of what I was communicating. While there may have been an element of paranoia / delusions of reference here, I’m inclined to think that this complex, multi-layered social interaction is the norm, but that we usually muddle through reasonably well letting most of it happen sub- or unconsciously. Conscious awareness of the complexity and breadth of it all can be enlightening, but also paralyzing.

It may be useful if you have hyperempathic tendencies to learn how to identify unwelcome empathy and put a damper on it.[95] Helen Riess of the Empathy and Relational Science Program of Boston’s Massachusetts General Hospital recommends exercises in deep breathing, detachment, and mindfulness as ways help you observe others without having your own reactivity or empathically-evoked emotions overwhelm you.

Meditation

There are varieties of meditation that seem to exercise empathy. The Mahayana Buddhist “exchanging self and others” meditation in which you assume the perspective of somebody else towards you is one example. Mettā meditation sometimes includes a component in which the meditator tries to imagine the basic desires of an expanding circle of beings. That sort of meditation is apparently effective at improving empathic accuracy.[96]

As an aside, there seems to be some convergence between the Buddhist no-self teaching and the Nietzschean critique of empathy mentioned earlier. In the Nietzschean point of view, when we empathize, we seek for a strong self in the other and project it onto them. Buddhism would agree that this projected self is an illusion, but would go further and say that we constitute our own illusory selves in much the same way. So Buddhist empathy may imply a peculiar psychology that is hard to compare to other forms.

Miscellaneous advice

Maybe you can learn and memorize certain universal facial expressions that can help you to gain empathic access to another person’s state-of-mind if you have difficulty doing so in a more immediately intuitive way.

Several authors suggested that you can improve your empathy by engaging in more cooperative pursuits with others.

Becoming more curious about the people around you (or being more bold about indulging the curiosity you already have) may help.

Being more aware of the biases that accompany empathy, and bringing those to mind when you find yourself empathizing, may help you to bring your empathic responses more in line with your values.

Conclusion

Empathy is something people do. We can do it in a way that is better or worse at contributing to our human flourishing. If we characteristically empathize well, we exhibit the virtue of empathy. This seems to be, as with other virtues, something we can learn and improve at with practice. I hope this post will help.

  1. ^

    Susan Lanzoni “Empathy’s Evolution in the Human Imagination” Zocalo Public Square 17 July 2017

  2. ^

    C. Daniel Batson Altruism in humans (2011)

  3. ^

    Fritz Breithaupt The Dark Sides of Empathy (2019)

  4. ^

    Paul Bloom Against Empathy: The Case for Rational Compassion (2016)

  5. ^

    Karla McLaren The Art of Empathy: A Complete Guide to Life’s Most Essential Skill (2013)

  6. ^

    Roman Krznaric Empathy: Why It Matters, and How to Get It (2014)

  7. ^

    Karla McLaren thinks the affective/cognitive description is misleading, though she also subdivides empathy in a similar way to people who use that primary division. Karla McLaren The Art of Empathy: A Complete Guide to Life’s Most Essential Skill (2013) pp. 26–42

  8. ^

    for example Paul Bloom (Against Empathy: The Case for Rational Compassion, 2016) says cognitive empathy isn’t empathy but “understanding,” while the empathy he’s Against is mostly the affective kind

  9. ^

    Daniel Goleman “What Is Empathy?” in Empathy Harvard Business Review Press (2007)

  10. ^

    Elizabeth A. Segal, et al. Assessing Empathy (2017)

  11. ^

    Adam Smith, The Theory of Moral Sentiments (1759)

  12. ^

    e.g. P.A. Miller & N. Eisenberg “The relation of empathy to aggressive and externalizing antisocial-behavior” Psychological Bulletin (1988) pp. 324–344

  13. ^

    Elizabeth A. Segal, et al. Assessing Empathy (2017) calls it “emotional egocentricity bias”.

  14. ^

    M. Toi, & C.D. Batson, “More evidence that empathy is a source of altruistic motivation” Journal of Personality and Social Psychology 43 (1982) pp. 281–292

  15. ^

    Paul Bloom Against Empathy: The Case for Rational Compassion (2016) chapter 2

  16. ^

    Paul Bloom Against Empathy: The Case for Rational Compassion (2016) chapter 4

  17. ^

    Elizabeth A. Segal, “Social empathy: A model built on empathy, contextual understanding, and social responsibility that promotes social justice” Journal of Social Service Research 37 (2011) pp. 266–267

  18. ^

    Elizabeth A. Segal, et al. “Developing the Social Empathy Index: An Exploratory Factor Analysis” Advances in Social Work 13 (2012)

  19. ^

    Elizabeth A. Segal, et al. Assessing Empathy (2017)

  20. ^

    Fritz Breithaupt The Dark Sides of Empathy (2019), chapter 2

  21. ^

    Sara H. Konrath, Edward H. O’Brien, & Courtney Hsing “Changes in Dispositional Empathy in American College Students Over Time: A Meta-Analysis” Personality and social psychology review 15 (2011)

  22. ^

    e.g. Maia Szalavitz “Shocker: Empathy Dropped 40% in College Students Since 2000” Psychology Today 28 May 2010

  23. ^

    Pierre Cérésole For Peace and Truth (1954)

  24. ^

    Martin Hoffman Empathy and Moral Development (2000)

  25. ^

    Jeremy Rifkin The Empathic Civilization: The race to global consciousness in a world in crisis (2009)

  26. ^

    Roman Krznaric Empathy: Why It Matters, and How to Get It (2014) pp. 178–179

  27. ^

    B.K.Y. Lau, J. Geipel, Y. Wu, & B. Keysar (2022). “The extreme illusion of understanding.” Journal of Experimental Psychology: General.

  28. ^

    Leslie Jamison The Empathy Exams (2014), p. 5

  29. ^

    Karla McLaren The Art of Empathy: A Complete Guide to Life’s Most Essential Skill (2013), p. 32

  30. ^

    Karla McLaren The Art of Empathy: A Complete Guide to Life’s Most Essential Skill (2013), pp. 188–189

  31. ^

    Roman Krznaric Empathy: Why It Matters, and How to Get It (2014) pp. 98+

  32. ^

    “The Multiplicity of Empathy: Study Reveals Both Interpersonal & Intrapersonal Benefits of Being Empathetic” (press release)

  33. ^

    R.B. van Baaren, R.W. Holland, K. Kawakami, & A. van Knippenberg “Mimicry and prosocial behavior” Psychological Science 15 (2004) pp. 71–74

    B.C.N. Müller, A.J. Maaskant, R.B. van Baaren, & A. Dijksterhuis “Prosocial consequences of imitation” Psychological Reports 110 (2012) pp. 891–898

    R.B. van Baaren, R. Janssen, T.L. Chartrand, & A. Dijksterhuis “Where is the love? The social aspects of mimicry.” Philosophical Transactions of the Royal Society Biological Sciences 364 (2009) pp. 2381–2389

  34. ^

    D. Grühn D, et al. “Empathy across the adult lifespan: Longitudinal and experience-sampling findings” Emotion 8 (2008) pp. 753–765

  35. ^

    Jason Kottke “Giving Is a Form of Attention” kottke.org 8 April 2022

  36. ^

    D. Grühn D, et al. “Empathy across the adult lifespan: Longitudinal and experience-sampling findings” Emotion 8 (2008) pp. 753–765

  37. ^

    B. Bernard Resiliency: What we have learned (2004)

  38. ^

    Bruce P. Doré et al. “Helping Others Regulate Emotion Predicts Increased Regulation of One’s Own Emotions and Decreased Symptoms of Depression” Personality & Social Psychology Bulletin 43 (2017) pp. 729–739

  39. ^

    Fritz Breithaupt The Dark Sides of Empathy (2019), Epilogue

  40. ^

    Paul Bloom Against Empathy: The Case for Rational Compassion (2016) chapter 6

  41. ^

    Leslie Jamison The Empathy Exams (2014), title essay

  42. ^

    Kaitlin Ugolik Phillips The Future of Feeling: Building Empathy in a Tech-Obsessed World (2020), pp. 101–102

  43. ^

    Jon Kolko “A Process for Empathetic Product Design” in Empathy Harvard Business Review Press (2007)

  44. ^

    Roman Krznaric Empathy: Why It Matters, and How to Get It (2014) pp.ⅺ–ⅻ

  45. ^

    Adam Waytz “The Limits of Empathy” in Empathy Harvard Business Review Press (2007)

  46. ^

    “Fidelity exploits virtual reality for ‘empathy training’ ” Finextra 16 October 2017

  47. ^

    Rolfe Winkler, Daisuke Wakabayashi, & Elizabeth Dwoskin “Apple Buys Artificial-Intelligence Startup Emotient” Wall Street Journal 7 January 2016. Quote: “…amassing an enormous database of human emotions using technology that relies on algorithms to analyze people’s faces and potentially discover their deepest feelings.”

  48. ^

    Harma Meffert, et al. “Reduced Spontaneous but Relatively Normal Deliberate Vicarious Representations in Psychopathy” Brain 136 (2013) pp. 2550–2562

  49. ^

    Jon Sutton, Peter K. Smith, & John Swettenham “Social Cognition and Bullying: Social Inadequacy or Skilled Manipulation?” British Journal of Developmental Psychology 17 (1999) pp. 435–450

  50. ^

    Eric W. Dolan “Psychopathic tendencies linked to reduced susceptibility to contagious yawning” Psypost 29 April 2022

  51. ^

    Roman Krznaric Empathy: Why It Matters, and How to Get It (2014) p.ⅹⅶ

  52. ^

    Roman Krznaric Empathy: Why It Matters, and How to Get It (2014) pp.ⅹⅶ–ⅹⅷ

  53. ^

    Simon Baron-Cohen The Science of Evil: On Empathy and the Origins of Cruelty (2012) p. 194

  54. ^

    Leslie Jamison The Empathy Exams (2014) pp. 4–5, 12

  55. ^

    Ezekiel Gleichgerrcht & Jean Decety “The Costs of Empathy among Health Professionals” in J. Decety, ed. (2012) Empathy: From bench to bedside p. 255

  56. ^

    J. Halbesleben, et al. “Too Engaged? A conservation of Resources View of the Relationships Between Work Engagement and Work Interference with Family” Journal of Applied Psychology 94.6 (2009)

  57. ^

    Cicero, Tusculan Disputations (~45BC), book Ⅲ “On Grief of Mind”

  58. ^

    for example, David D. Vachon, Donald R. Lynam, & Jarrod A. Johnson “The (Non) Relation between Empathy and Aggression: Surprising Results from a Meta-Analysis” Psychological Bulletin 140 (2014) p. 16

  59. ^

    Betsy Blaney “Baby Jessica turns 25, gains access to trust fund” Associated Press 26 March 2011

  60. ^

    C.D. Batson, T.R. Klein, L. Highberger, & L.L. Shaw “Immorality from empathy-induced altruism: When compassion and justice conflict” Journal of Personality and Social Psychology 68 (1995) pp. 1042–1054

    Anneke E.K. Buffone & Michael J. Poulin “Empathy, Target Distress, and Neurohormone Genes Interact to Predict Aggression for Others—Even Without Provocation” Personality and Social Psychology Bulletin 40 (2014) pp. 1406–1422

  61. ^

    Michael N. Stagnaro & Paul Bloom “The Paradoxical Effects of Empathy on the Willingness to Punish” (unpublished, 2016)

  62. ^

    S.G. Young & K. Hugenberg “Mere socialization categorization modulates identification of facial expressions of emotion” Journal of Personality and Social Psychology 99 (2010) pp. 964–977

  63. ^

    description of (not quote from) J.J. Van Bavel & W.A. Cunningham “Self-categorization with a novel mixed-race group moderates automatic social and racial biases” Personality and Social Psychology Bulletin 35 (2009) pp. 321–335 [description from Elizabeth A. Segal, et al. Assessing Empathy (2017)]

  64. ^

    M. Cikara & S.T. Fiske “Bounded empathy: Neural responses to outgroup targets’ (mis)fortunes” Journal of Cognitive Neuroscience 23 (2011) pp. 3791–3803

    G. Bucchioni & al. “Do we feel the same empathy for loved and hated peers?” PLoS ONE 10 (2015) pp. 1–11

  65. ^

    M.W. Kraus, S. Côté, & D. Keltner “Social class, contextualism, and empathic accuracy” Psychological Science 2 (2010) pp. 1716–1723

  66. ^

    J. Hogeveen, M. Inzlicht, & S.S. Obhi “Power changes how the brain responds to others” Journal of Experimental Psychology 143 (2014) pp. 755–762

    A.D. Galinksy et al. “Power and perspectives not taken” Psychological Science 17 (2006) pp. 1068–1074

    G.A. Van Kleef et al. “Power gets you high: The powerful are more inspired by themselves than by others” Social Psychology and Personality Science 6 (2015) pp. 472–480

  67. ^

    Elizabeth A. Segal, et al. Assessing Empathy (2017) ch. 4; Stéphane Côté, et al. “Social power facilitates the effect of prosocial orientation on empathic accuracy” Journal of Personality and Social Psychology 101 (2011) pp. 217–232

  68. ^

    M. Schmid Mast & A. Darioly “Emotion recognition accuracy in hierarchical relationships” Swiss Journal of Psychology 73 (2014) pp. 69–75

  69. ^

    Christopher J. Einolf “Empathic concern and prosocial behaviors: A test of experimental results using survey data” Social Science Research 37 (2008) pp. 1267–1279

  70. ^

    J. Decety, S. Echols, & J. Correll “The blame game: The effect of responsibility and social stigma on empathy for pain” Journal of Cognitive Neuroscience 22 (2009) pp. 985–997

  71. ^

    Paul Bloom Against Empathy: The Case for Rational Compassion (2016) chapter 5

  72. ^

    Fritz Breithaupt The Dark Sides of Empathy (2019), chapter 5

  73. ^

    Fritz Breithaupt The Dark Sides of Empathy (2019), chapter 4

  74. ^

    Fritz Breithaupt The Dark Sides of Empathy (2019)

    Friedrich Nietzsche Beyond Good and Evil (1886) and The Geneaology of Morals (1887)

  75. ^

    E. Teding van Berkhout & J.M. Malouff “The efficacy of empathy training: A meta-analysis of randomized controlled trials” Journal of Counseling Psychology 63 (2016) pp. 32–41

  76. ^

    Karla McLaren The Art of Empathy: A Complete Guide to Life’s Most Essential Skill (2013), chapter 8

  77. ^

    Karla McLaren The Art of Empathy: A Complete Guide to Life’s Most Essential Skill (2013), p. 71

  78. ^

    Karla McLaren The Art of Empathy: A Complete Guide to Life’s Most Essential Skill (2013), pp. 236–261

  79. ^

    George Eliot “The Natural History of German Life” Westminister Review ⅬⅩⅥ (July 1856) pp. 28–44

  80. ^

    Steven Pinker The Better Angels of our Nature: Why Violence Has Declined (2011) p. 175

  81. ^

    David Dodell-Feder, et al. “Fiction reading has a small positive impact on social cognition: A meta-analysis” Journal of Experimental Psychology: General 147 (2018), pp. 1713–1727

  82. ^

    Lynn Hunt Inventing Human Rights: A History (2007)

  83. ^

    Lynn Hunt “Inventing Human Rights” (lecture, March 2008)

  84. ^

    Karla McLaren The Art of Empathy: A Complete Guide to Life’s Most Essential Skill (2013) chapter 6 

  85. ^

    Rachel Ruttan, Mary-Hunter McDonnell, & Loran Nordgren “It’s Harder to Empathize with People If You’ve Been in Their Shoes” in Empathy Harvard Business Review Press (2007)

    Rachel L. Ruttan, et al. “Having ‘been there’ doesn’t mean I care: when prior experience reduces compassion for emotional distress.” Journal of Personality and Social Psychology 108 (2015) pp. 610–622 

  86. ^

    Elizabeth A. Segal, et al. Assessing Empathy (2017), chapter 4

  87. ^

    C.D. Batson, et al. “Empathy and attitudes: can feeling for a member of a stigmatized group improve feelings toward the group?” Journal of Personality and Social Psychology 72 (1997) pp. 105–118

  88. ^

    T. Saguy, et al. “Awareness of intergroup help can rehumanize the out-group” Social Psychological and Personality Science 6 (2015) pp. 551–558

  89. ^

    Keith C. Barton & Alan W. McCully “Trying to ‘See Things Differently:’ Northern Ireland Students’ Struggle to Understand Alternative Historical Perspectives” Theory & Research in Social Education 40 (2012)

  90. ^

    Scott Davis e.g. “Tennis legend Andre Agassi revealed that he learned how to beat a rival by watching his tongue on serves” Yahoo! Sports 29 April 2021

  91. ^

    Liz Stinson “Can a Card Game Teach You Empathy? This Creative Agency Thinks So” Eye on Design 26 May 2017

  92. ^

    Ben Davis “Can VR Really Make Us Feel Empathy? Alejandro G. Iñárritu’s ‘Carne y Arena’ Proves That’s the Wrong Question” Artnet News 30 March 2018

  93. ^

    e.g. “A video game can change the brain, may improve empathy in middle schoolers” ScienceDaily 9 August 2018

  94. ^

    Kelly Doherty “The gaming industry is finally teaching players empathy” i-D 7 October 2021

  95. ^

    As I was putting the finishing touches on this post, Joanna Cannon’s “‘I feel your pain’: confessions of a hyper-empath” was published in the Guardian; it has a section with some advice on how to cope with hyperempathy.

  96. ^

    J.S. Mascaro, J.K. Rilling, L.T. Negi, & C.L. Raison “Compassion meditation enhances empathic accuracy and related neural activity” Social Cognitive and Affective Neuroscience 8 (2013), pp. 48–55



Discuss

Looking for someone to run an online seminar on human learning

3 мая, 2022 - 04:54
Published on May 3, 2022 1:54 AM GMT

I'm looking for someone with a background in education and/or cognitive science to run an online seminar for non-rationalists on how humans learn things and how to efficiently teach a subject to others. A few examples of the sort of content I'm thinking of are: Ebbinghaus's research on memory, spaced repetition, the difference between shallow and deep learning of a subject. The exact content would be up to you.

It would be a ~1 hour seminar on May 29th, run via Zoom or a similar platform. It pays $400 USD. If you're interested, please email me to discuss the details.



Discuss

Home Antigen Tests Aren’t Useful For Covid Screening

3 мая, 2022 - 04:30
Published on May 3, 2022 1:30 AM GMT

Epistemic status: I strongly believe this is the right conclusion given the available data. The best available data is not that good, and if better data comes out I reserve the right to change my opinion.

EDIT (4/27): In a development I consider deeply frustrating but probably ultimately good, the same office is now getting much more useful information from antigen tests. They aren’t tracking with same rigor so I can’t comapre results, but they are now beating the bar of “literally every noticing covid”.

In an attempt to avoid covid without being miserable, many of my friends are hosting group events but requiring attendees to take a home covid test beforehand. Based on data from a medium-sized office, I believe testing for covid with the tests people are using, to be security theater and provide no decrease in riskAntigen tests don’t work for covid screening. There is a more expensive home test available that provides some value, and rapid PCR may still be viable.

It’s important to distinguish between test types here: antigen tests look for viral proteins, and genetic amplification tests amplify viral RNA until it reaches detectable levels. The latter are much more sensitive. Most home tests are antigen tests, with the exception of Cue, which uses NAAT (a type of genetic amplification). An office in the bay area used aggressive testing with both Cue and antigen tests to control covid in the office and kept meticulous notes, which they were kind enough to share with me. Here are the aggregated numbers: 

  • The office requested daily Cue tests from workers. I don’t know how many people this ultimately included, probably low hundreds? I expect compliance was >95% but not perfect.
    • The results are from January when the dominant strain was Omicron classic, but no one got strain tested.
  • 39 people had at least one positive Cue test, all of which were either asymptomatic or ambiguously symptomatic (e.g. symptoms could be explained by allergies) at the time, and 27 of which had recent negative cue tests (often but not always the day before, sometimes the same day)
  • Of these, 10 definitely went on to develop symptoms, 7 definitely did not, and 18 were ambiguous (and a few were missing data).
  • 33 people with positives were retested with cue tests, of which 9 were positive. 
  • Of those 24 who tested positive and then negative, 4 tested positive on tests 3 or 4.
  • Of the 20 people with a single positive test followed by multiple negative retests, 6 went on to develop symptoms.
  • 0 people tested positive on antigen tests. There was not a single positive antigen test across this group. They not only didn’t catch covid as early as Cue did, they did not catch any cases at all, including at least 2 people who took the tests while experiencing definitive systems.
    • Antigen tests were a mix of Binax and QuickVue.
    • Early cases took multiple antigen tests over several days, later cases stopped bothering entirely.
    • The “negative test while symptomatic” count is artificially low because I excluded people with ambiguous symptoms, and because later infectees didn’t bother with antigen tests. 
    • I suppose I can’t rule out the possibility that they had an unrelated disease with similar symptoms and a false positive on the Cue test. But it seems unlikely that that happened 10-28 times out a few hundred people without leaving other evidence.

A common defense of antigen tests is that they detect whether you’re contagious at that moment, not whether you will eventually become contagious. Given the existence of people who tested antigen-negative while Cue-positive and symptomatic, I can’t take that seriously.

Unfortunately Cue tests are very expensive. You need a dedicated reader, which is $250, and tests are $65 each (some discount if you sign up for a subscription). A reader can only run 1 test at a time and each test takes 30 minutes, so you need a lot for large gatherings even if people stagger their entrances. 

My contact’s best guess is that the aggressive testing reduced but did not eliminate in-office spread, but it’s hard to quantify because any given case could have been caught outside the office, and because they were trying so many interventions at once. Multiple people tested positive, took a second test right away, and got a negative result, some of whom went on to develop symptoms; we should probably assume the same chance of someone testing negative when a second test would have come back positive, and some of those would have been true positives. So even extremely aggressive testing has gaps.

Meanwhile, have I mentioned lately how good open windows and air purifiers are for covid? And other illnesses, and pollution? And that taping a HEPA filter to a box fan is a reasonable substitute for an air purifier achievable for a very small number of dollars? Have you changed your filter recently? 

PS. Before you throw your antigen tests out, note that they are more useful than Cue tests for determining if you’re over covid. Like PCR, NAAT can continue to pick up dead RNA for days, maybe weeks, after you have cleared the infection. A negative antigen test after symptoms have abated and there has been at least one positive test is still useful evidence to me. 

PPS. I went through some notes and back in September I estimated that antigen testing would catch 25-70% of presymptomatic covid cases. Omicron moves faster, maybe faster enough that 25% was reasonable for delta, but 70% looks obviously too high now. 

PPPS. Talked to another person at the office, their take is the Cue tests are oversensitive. I think this fits the data worse but feel obliged to pass it on since they were there and I wasn’t.

PPPPS (5/02): multiple people responded across platforms that they had gotten positive antigen tests. One or two of these was even presymptomatic. I acknowledge the existence proof but will not be updating until the data has a denominator. If you’re doing a large event like a conference I encourage you to give everyone both cue, antigen, and rapid PCR tests and record their results, and who eventually gets sick. If you’d like help designing this experiment in more detail please reach out (elizabeth-at-acesounderglass.com)



Discuss

Is evolutionary influence the mesa objective that we're interested in?

3 мая, 2022 - 04:18
Published on May 3, 2022 1:18 AM GMT

Epistemic status: this is the result of me trying to better understand the idea of mesa optimizers. It's speculative and full of gaps, but maybe it's interesting and I'm not realistically going to have time to improve it much in the near future.

Humans are often presented as an example of "mesa optimisers" - organisms created to "maximise evolutionary fitness" that end up doing all sorts of other things including not maximising evolutionary fitness and transforming the world in the process. This analogy is usually accompanied by a disclaimer like this:

We do not expect our analogy to live up to intense scrutiny. We present it as nothing more than that: an evocative analogy (and, to some extent, an existence proof) that explains the key concepts.

I am proposing that if we focus on evolutionary "influence" instead of "fitness", we can flip both claims on their head:

  • Humans are extremely evolutionarily influential
  • We should take the evolution analogy seriously
Evolution is about how things change

I think evolutionary fitness is to some extent not the interesting thing about evolution. If the world was always full of indistinguishable rabbits and always will be full of indistinguishable rabbits, then rabbits are in some sense "fit", but evolution is also a boring theory because all it says is "rabbits". The interesting content of evolution is in what it says about how things change: if the world is full of regular rabbits plus one extra-fit rabbit, then evolution says that in a few years, the world will have few regular rabbits and lots of extra-fit rabbits.

 I want to propose a rough definition of evolutionary influence that generalises this idea. There are a few gaps in the definition which I hope can be successfully resolved, but I haven't had the time to do this yet.

First, we need an environment. I currently think of an environment as "a universe at a particular point in time". The universe is:

  • A set T.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')}  of "points in time"
  • A set Ω of configurations that the universe can have at a particular time
  • An update rule f:Ω→Δ(Ω) that probabilistically maps the configuration at one point in time to the configuration at the next (Δ(Ω) means "the set of probability distributions on Ω").

Given a time t∈T, the environment μt is a probability distribution on Ω.

An organism is a particular configuration of a "small piece" of a universe. We can specify a function  u:Ω→{0,1} that evaluates whether the universe contains the organism, and u is somehow restricted to evaluating "small parts" of the universe (what I mean by a "small part" is currently a gap in the definition). We can condition an environment μt on the presence of an organism u to get the environment μt|u and similarly μt|∼u is the environment without the organism u.

A feature is a "large piece" of an environment. Like organisms, I'm not sure what I mean by "large piece". In any case, there's a function v:Ω→{0,1} that tells us whether a feature is present, and it must in some sense be "big and obvious".

An organism u at t has a large evolutionary influence at t">t′>t if the probability of some large feature v is very different in the future environment with the organism μt|uft′−t than in the future environment without it μt|∼uft′−t

Intuition: If at time t the environment μt is  full of grass and there are also a pair of rabbits u there, then in the future t′ the environment μt|uft′−t will be full of rabbits and have a lot less grass. On the other hand, if there are no rabbits at time t then the future environment μt|∼uft′−t will still be mostly grass.

Intuition 2: Perhaps if humans had not appeared when they did, highly intelligent life would have taken much longer to appear on Earth/never done so.  Then the Earth without humans 300k years ago wouldn't have any cities etc. today.

Relevance to AI

The fundamental question I'm asking here is: are AI research efforts likely to produce highly influential "organisms". A second important question is whether this influence is aligned with the creator's aims, but this seems to me to add a lot of complication.

My basic thinking here is that an AI system in training is embedded in two "universes". Think of a large neural network in training. One "universe" in which it lives is the space of network weights, and the update rule is given by the training algorithm and the loss incurred on the data at each step of training. It's not clear that it's meaningful to talk about "influentialness" in this universe. Maybe there is some "small" feature of the initialisation that determines whether it converges to something useful or not, but that is speculative (and I don't know what I mean by "small").

It also exists in the real universe - i.e. it's a configuration in the storage of some computer somewhere. Here there's a more intuitive sense that we can talk about "influentialness" - if it produces useful outputs somehow, people will be excited by their new AI and publish papers about it, create products using it and so forth, whereas if it doesn't then none of that will happen and it will be forgotten.

Given the way neural networks are trained, a trained neural network basically has to be something that performs reasonably well in the training universe. However, influentialness in the real universe trumps performance in the training universe - a real-universe-influential AI that isn't actually good on the training set is still real-universe-influential.

Two postulates about influentialness
  1. There is a training universe that, when run for long enough, produces a highly influential organism in our universe
    • An example of this would be a very high-performance reinforcement learner whose reward is based on some "large feature" of our universe
  2. AI training that we actually do has a reasonable chance of creating such a training universe
    • For example, maybe it's not too hard to repurpose existing techniques (or future developments of them) to create the reinforcement learner mentioned in 1

Speculatively, there is some sense in which we can talk about the compatibility of the training environment and the real universe in the sense that "high performance" in the training environment is correlated with influence in the real universe. For a hypothetically "optimal" reinforcement learner rewarded based on large features of the real universe, this compatibility is maximal. However, even more pragmatic AI training regimes might exhibit high compatibility, and it also might not be so transparent about what kind of influence it is likely to exert. In particular, influence trumps performance on the training objective, so good behaviour assuming optimality with respect to the training objective cannot necessarily be relied on, and this seems to me to be one of the key insights of the idea of mesa-optimisation. On the other hand, it's completely plausible to me that good behaviour assuming optimality could imply good behaviour for near-optimality too. It is also quite mysterious to me how to actually characterise "good behaviour".



Discuss

Open & Welcome Thread - May 2022

3 мая, 2022 - 02:47
Published on May 2, 2022 11:47 PM GMT

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the new Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.



Discuss

Predicting for charity

3 мая, 2022 - 01:59
Published on May 2, 2022 10:59 PM GMT

Excerpted from Above the Fold.

Prediction markets succeed when they require people to bet something they value, like money. But past attempts at real-money prediction markets like Intrade have been shut down by the CFTC. At Manifold Markets, we currently allow users to trade on Manifold Dollars (aka M$ or “mana”), an in-game currency specific to our platform. But one of the most common criticisms we hear is: “Why should I care about trading fake currency?”

It would be nice to find a middle ground where users can bet something of real value which doesn’t run afoul of financial regulations. Thinking about this, we were inspired by donor lotteries: if you can gamble with charitable donations, shouldn’t you be able to make bets with them? Thus, Manifold for Good was born.

You can now donate your M$ winnings to charity! Through the month of May, every M$ 100 you contribute turns into USD $1 sent to your chosen charity - we’ll cover all processing fees!

Why?

Manifold for Good solves two problems with today’s prediction markets. First, it allows you to bet using something valuable to you (i.e. donations to your favorite charity), which increases the incentive to bet correctly, relative to just virtual points. Second, it respects existing financial regulations, which has proven difficult for prediction markets in the past.

By providing an entertaining and impactful way to allocate money to charity, Manifold for Good can also increase the total amount of money donated. Just as donors participating in a charity bingo night are willing to pay extra for the value of entertainment, so too can Manifold’s markets provide a fun, motivating reason to participate in charitable activities.

What’s Next?

Think of Manifold for Good as an experiment! We’re seeing what the level of demand is for this kind of redemption for Manifold Dollars; let us know if you have any thoughts or suggestions.

In the future, we’d like to grow the program to increase the number of available charities. We currently support 30+ charities; if you have a charity recommendation, let us know and we’ll pay a M$ 500+ bounty once we add it!

We’d also love to offer donation matching to cause areas or charities — we think this would get users even more excited about forecasting and donating! If you would like to partner with us to fund an experiment like this, or be featured as a charity, please get in touch at give@manifold.markets.

Finally: one HUGE shoutout to Sam Harsimony and Sinclair Chen for leading the effort to build out Manifold for Good!

Note: we are not affiliated with most of these charities, other than being fans of their work. As Manifold itself is a for-profit org, your M$ contributions will not be tax-deductible.

Bonus: our codebase is now open source!

At Manifold, we’ve always aimed to be transparent about the way we do things. Some examples:

I’m excited to say that we’ve taken the next step forward: open sourcing our entire codebase! Check out our Github repo here.

Don’t forget to like and subscribe leave a Github Star!

This effort was spearheaded by Marshall Polaris, to whom the whole Manifold community owes a huge debt of gratitude. Marshall has been behind the scenes, doing the necessary work to prepare us for this big step forward:

  • Ensuring our user data is secure against attackers
  • Expanding our documentation to help new contributors get up and running
  • Improving our processes to scale up with many new potential contributors

I can’t wait to see what features you build, bugs you fix, or projects you start, now that our code is open to you!

— Austin

Kudos to Justis Millis for reviewing a draft of this post.



Discuss

Information security considerations for AI and the long term future

2 мая, 2022 - 23:54
Published on May 2, 2022 8:54 PM GMT

Summary

This post is authored by Jeffrey Ladish, who works on the security team at Anthropic, and Lennart Heim, who works on AI Governance with GovAI (more about us at the end). The views in the post are our own and do not speak for Anthropic or GovAI. This post follows up on Claire Zabel and Luke Muehlhauser’s 2019 post, Information security careers for GCR reduction.

We’d like to provide a brief overview on:

  1. How information security might impact the long term future
  2. Why we’d like the community to prioritize information security

In a following post, we will explore:

  1. How you could orient your career toward working on security

Tl;dr: 

  • New technologies under development, most notably artificial general intelligence (AGI), could pose an existential threat to humanity. We expect significant competitive pressure around the development of AGI, including a significant amount of interest from state actors. As such, there is a large risk that advanced threat actors will hack organizations — that either develop AGI, provide critical supplies to AGI companies, or possess strategically relevant information— to gain a competitive edge in AGI development. Limiting the ability of advanced threat actors to compromise organizations working on AGI development and their suppliers could reduce existential risk by decreasing competitive pressures for AGI orgs and making it harder for incautious or uncooperative actors to develop AGI systems.
What is the relevance of information security to the long term future?

The bulk of existential risk likely stems from technologies humans can develop. Among candidate technologies, we think that AGI, and to a lesser extent biotechnology, are most likely to cause human extinction. Among technologies that pose an existential threat, AGI is unique in that it has the potential to permanently shift the risk landscape and enable a stable future without significant risks of extinction or other permanent disasters. While experts in the field have significant disagreements about how to navigate the path to powerful aligned AGI responsibly, they tend to agree that actors that seek to develop AGI should be extremely cautious in the development, testing, and deployment process, given the failures could result in catastrophic risks, including human extinction.

NIST defines information security as “The protection of information and information systems from unauthorized access, use, disclosure, disruption, modification, or destruction.” 

We believe that safe paths to aligned AGI will require extraordinary information security effort for the following reasons:

  • Insufficiently responsible or malicious actors are likely to target organizations developing AGI, software and hardware suppliers, and supporting organizations to gain a competitive advantage.
  • Thus, protecting those systems will reduce the risk that powerful AGI systems are developed by incautious actors hacking other groups.

Okay, but why is an extraordinary effort required?

  • Plausible paths to AGI, especially if they look like existing AI systems, expose a huge amount of attack surface because they’re built using complex computing systems with very expansive software and hardware supply chains.
  • Securing systems as complex as AGI systems is extremely difficult, and most attempts to do this have failed in the past, even when the stakes have been large, for example, the Manhattan Project.
  • The difficulty of defending a system depends on the threat model, namely the resources an attacker brings to bear to target a system. Organizations developing AGI are likely to be targeted by advanced state actors who are amongst the most capable hackers.

Even though this is a challenging problem requiring extraordinary effort, we think the investments are worth pursuing, and the current measures are insufficient.

What does it mean to protect an AI system?

It can be helpful to imagine concrete scenarios when thinking about AI security — the security of AI systems. Gwern recently wrote a compelling fictional take on one such scenario, including some plausible infosec failures, which we recommend checking out. 

Infosec people often think about systems in terms of attack surface. How complex is the system, how many components does it have, how attackable is each component, etc? Developing a modern AI system like GPT-3 involves a lot of researchers and developers -- the GPT-3 paper had 31 authors! Each developer has their own laptop, and needs some amount of access to the models they work on. Usually, code runs on cloud infrastructure, and there are many components that make up that infrastructure. Data needs to be collected and cleaned. Researchers need systems for training and testing models. In the case of GPT-3, an API is created to grant limited access for people outside the company. 

Figure 1: Overview of the active components in the development of an ML system. Each introduces more complexity, expands the threat model, and introduces more potential vulnerabilities.

Most of the components described here contain a staggering amount of complexity (see Figure 1). For example, the Linux kernel alone contains over 20 million lines of code. Each piece of hardware and software is a component that could be exploited. The developer could be using a malicious browser plugin that steals source code, or their antivirus software could be silently exfiltrating critical information. The underlying cloud infrastructure could be compromised, either because of underlying exploits in the cloud provider or because of misconfigurations or vulnerable software introduced by the organization.

And these are just the technical components. What’s not depicted are the humans creating and using the software and hardware systems. Human operators are generally the weakest part of a computing system. For example, developers or system administrators could be phished, phones used for multifactor authentication systems could be SIM-swapped, or passwords for key accounts could be reset by exploiting help desk employees. This is generally described as social engineering. In short, modern AI systems are very complex systems built by many people, and thus are fundamentally difficult to secure.

Threat Actors and AGI

Securing a modern AI project against attacks from general cybercriminals — such as ransomware operators, extortionists, etc —is difficult but not extraordinarily difficult. Securing a system as complex as a modern AI system against a state actor or an Advanced Persistent Threat (APT) actor is extremely difficult, as the examples in the reference class failures section demonstrate. We believe it is quite likely that state actors will increasingly target organizations developing AGI for the reasons listed below:

  • Near-term AI capabilities may give states significant military advantages
  • State-of-the-art AI labs have valuable IP and are thus a rich target for state-sponsored industrial espionage
  • Concepts of the strategic value of AGI are present and accessible in our current culture

The US, China, Russia, Israel, and many other states are currently acting as if modern and near-term AI capabilities have the potential to improve strategic weapons systems, including unmanned aerial vehicles (UAVs), unmanned underwater vehicles (UUVs), submarine detection, missile detection systems, and hacking tools.

Some of these areas have already been the subject of successful state hacking activities. Open Philanthropy Project researcher Luke Muelhauser compiles several examples in this document, including: “In 2011-2013, Chinese hackers targeted more than 100 U.S. drone companies, including major defense contractors, and stole designs and other information related to drone technology…  Pasternack (2013); Ray (2017).”

It’s not evident that espionage aimed at strategic AI technologies will necessarily target labs working on AGI if AGI itself is not recognized as a strategic technology, perhaps because it is perceived as too far out to be beneficial. However, it seems likely that AGI labs will develop more useful applications as they get closer to capable AGI systems, and that some of these applications will touch on areas states care about. For example, China used automated propaganda to target Taiwan elections in 2018 and could plausibly be interested in stealing language models like GPT-3 for this purpose. Code generation technologies like Codex and AlphaCode might have military applications as well, perhaps as hacking tools.

In addition to the direct military utility of AI systems, state-sponsored espionage is also likely to occur for the purpose of economic competition. AI companies have raised $216.6B in investment, with at least a couple of billion raised by companies specifically trying to pursue AGI development which makes them a valuable economic target. As already outlined, stealing IP for AGI development itself or required critical resources is of interest to malicious actors, such as nations or companies. If IP violations are hard to detect or enforce, it makes industrial espionage especially attractive.

The most concerning reason that states may target AGI labs is that they may believe that AGI is a strategically important technology. Militaries and intelligence agencies have access to the same arguments that convinced the EA community that AI risk is important. Nick Bostrom discusses the potential strategic implications of AGI in Superintelligence (2014, ch. 5).  Like with the early development of nuclear weapons, states may fear that their rivals might develop AGI before them.

Note that it is the perception of state actors which matters in these cases. States may perceive new AI capabilities as having strategic benefits even if they do not. Likewise, if AI technologies have the potential for strategic impact but state actors do not recognize this, then state espionage is less of a concern.

Other targets

Next to targeting organizations that develop AGI systems, we also think that organizations that either (a) supply critical resources to AGI labs or (b) research and develop AGI governance strategies are also at risk:

  • Suppliers of critical resources, such as compute infrastructure or software, are of significant relevance for AGI organizations. These supplies are often coming from external vendors. Targeting vendors is a common practice for state actors trying to gain access to high value targets. In addition, stealing intellectual property (IP) from external vendors could boost a malicious actor's capabilities of developing relevant resources (which it might have previously been excluded from due to export bans).
  • We think that AI Governance researchers may also be targets if they are perceived to possess strategic information about AI systems in development. State actors are likely especially interested in obtaining intelligence on national organizations and those relevant to the military.
Reference Class Failures

In trying to understand the difficulty of securing AI systems, it is useful to look at notable failures in securing critical information and ways to mitigate them. There is no shortage of examples — for a more comprehensive list, see this shared list of examples of high-stake information security breaches by Luke Muehlhauser.  We want to present some recent examples — walking on the spectrum from highly technical state-sponsored attacks to relatively simple but devastating social engineering attacks. 

One recent example of a highly technical attack is the Pegasus 0-click exploit. Developed by the NSO group, a software firm that sells “technology to combat terror and crime to governments only”, this attack enabled actors to gain full control of the victim’s iPhone (reading messages, files, eavesdropping, etc.). It was used to spy on human rights activists and was also connected to the murder of Jamal Khashoggi. As outlined in this blogpost by Google’s Project Zero, this attack was highly sophisticated and required a significant amount of resources for its development — costing millions of dollars for state actors to purchase.

On the other side of the spectrum, we have the Twitter account hijacking in 2020, where hackers gained access to internal Twitter tools by manipulating a small number of employees to gain access to their credentials. This then allowed them to take over Twitter accounts of prominent figures, such as Elon Musk and Bill Gates — and all of this was probably done by some teenagers.

Another more recent attack, which also likely relied heavily on social engineering tactics, is the hack of NVIDIA by the Lapsus$ group. This attack is especially interesting as NVIDIA is an important actor in developing AI chips. Their intellectual property (IP) being leaked and potentially being used by less cautious actors could accelerate competitors’ efforts in developing more powerful AI chips while actively violating others' IP norms. Notably, while the target of the hacking group is a target of interest to state actors, we have some first hints that this attack might actually have been conducted by a number of teenagers.

The Twitter account hijacking and recent NVIDIA hack are notable as the required resources for those attacks were relatively small. More significant efforts and good security practices could have mitigated those attacks or at least made them significantly more expensive. Companies that are of strategic importance, or in respect to this article, organizations relevant to the development of AGI systems, should benefit from the best security and be on a similar security level, as national governments — given their importance for (national) security.

The applicability of the security mindset to alignment work

Good engineering involves thinking about how things can be made to work; the security mindset involves thinking about how things can be made to fail.

Bruce Schneier

In addition to securing information that could play an outsized role in the future of humanity, we think that the information security field also showcases ways of thinking that are essential for AI alignment efforts. These ideas are outlined in Eliezer Yudkowsky’s post Security Mindset and Ordinary Paranoia and follow up post Security Mindset and the Logistic Success Curve. The security mindset is the practice of looking at a system through the lens of adversarial optimization, not just looking for ways to exploit the system, but looking for systemic weakness that might be abused even if the path to exploitation is not obvious. In the second post, Eliezer describes the kind of scenario where the security mindset is crucial to building a safe system:

…this scenario might hold generally wherever we demand robustness of a complex system that is being subjected to strong external or internal optimization pressures… Pressures that strongly promote the probabilities of particular states of affairs via optimization that searches across a large and complex state space…Pressures which therefore in turn subject other subparts of the system to selection for weird states and previously unenvisioned execution paths… Especially if some of these pressures may be in some sense creative and find states of the system or environment that surprise us or violate our surface generalizations…

This scenario describes security critical code like operating system kernels, and even more so describes AGI systems. AGI systems are incredibly dangerous because they are powerful, opaque optimizers and human engineers are unlikely to have a good understanding of the scope, scale, or target of their optimization power as they are being created. We believe this task is likely to fail without a serious application of the security mindset.

Applying the security mindset means going beyond merely imagining ways your system might fail. For example, if you’re concerned your code-writing model-with-uncertain-capabilities might constitute a security threat, and you decide that isolating it in a Docker container is sufficient, then you have failed to apply the security mindset. If your interpretability tools keep showing your models say things its internal representation knows are false, and you decide to use the output of the interpretability tools to train the model to avoid falsehoods, then you have failed to apply the security mindset.

The security mindset is hard to learn and harder to teach. Still, we think that more interplay between the information security community and the AI alignment community could help more AI researchers build skills in this area and improve the security culture of AI organizations.

Building the relevant information security infrastructure now

We think that labs developing powerful AGI systems should prioritize building secure information systems to protect against the theft and abuse of these systems. In addition, we also think that other relevant actors, such as organizations working on strategic research or critical suppliers, are increasingly becoming a target and also need to invest in information security controls.

Our appeal to the AI alignment and governance community is to take information security seriously now, in order to build a firm foundation as threats intensify. Security is not a feature that’s easy to tack on later. There is ample low-hanging fruit — using up to date devices and software, end-to-end encryption, strong multi-factor authentication, etc. Setting up these controls is an excellent first step and high investment return. However, robust information security controls will require investment and difficult tradeoffs. People are usually the weak point of information systems. Therefore, training and background checks are essential. 

Information security needs are likely to become much more demanding as AGI labs, and those associated with AGI labs, are targeted by increasingly persistent and sophisticated attacks. In the recent Lapsus$ attacks, personal accounts were often targeted to gain access to 2FA systems to compromise company accounts. Tools like Pegasus, already utilized by agencies in dozens of countries, could easily be leveraged against those working on AI policy and research.

While we could make many more specific security recommendations, we want to emphasize the importance of threat awareness and investment in secure infrastructure, rather than any specific control. That being said, if you do have questions about specific security controls, or want help making your org more secure, feel free to reach out! Part of our goal is to help organizations just starting to think about security get started, as well as helping existing organizations to ramp up their security programs.

Conclusion

In this post, we’ve presented our argument for the importance of information security for the long term future. In the next post, we’ll give some concrete suggestions for ways people could contribute to the problem, including: 

  • 1) how to know if an infosec career is a good idea for you
  • 2) how to orient your career toward information security and
  • 3) how others working to reduce AI risk can acquire and incorporate infosec skills into their existing work

In the meantime, you can engage with others on related discussions in the Information Security in Effective Altruism Facebook group.

About us
  • About Jeffrey
    • I work on the security team at Anthropic (we’re hiring!), and am also working on AI security field building and strategy. I've worn several hats in my infosec career: I worked as a security engineer at Concur Technologies,  led security efforts for the cryptocurrency company, Reserve, and then started my own security consultancy business. I also spent a couple of years exploring existential and catastrophic risks from nuclear weapons and biotechnology. You can find some of my work here: https://jeffreyladish.com
  • About Lennart
    • I work on the intersection of AI hardware and AI Governance. Before that, I studied Computer Engineering and have a long-standing interest in security (though I never worked professionally full-time in this field). I used to work on security as a research assistant in wireless and networked systems or in my leisure time, mostly on embedded systems and webservers.
Acknowledgements

Many thanks to Ben Mann, Luke Muehlhauser, Markus Anderljung, and Leah McCuan for feedback on this post.



Discuss

My thoughts on nanotechnology strategy research as an EA cause area

2 мая, 2022 - 21:07
Published on May 2, 2022 5:57 PM GMT

This is a cross-post from the Effective Altruism Forum.

Two-sentence summary: Advanced nanotechnology might arrive in the next couple of decades (my wild guess: there’s a 1-2% chance in the absence of transformative AI) and could have very positive or very negative implications for existential risk. There has been relatively little high-quality thinking on how to make the arrival of advanced nanotechnology go well, and I think there should be more work in this area (very tentatively, I suggest we want 2-3 people spending at least 50% of their time on this by 3 years from now).

Context: This post reflects my current views as someone with a relevant PhD who has thought about this topic on and off for roughly the past 20 months (something like 9 months FTE). Note that some of the framings and definitions provided in this post are quite tentative, in the sense that I’m not at all sure that they will continue to seem like the most useful framings and definitions in the future. Some other parts of this post are also very tentative, and are hopefully appropriately flagged as such.

Key points
  • I define advanced nanotechnology as any highly advanced future technology, including atomically precise manufacturing (APM), that uses nanoscale machinery to finely image and control processes at the nanoscale, and is capable of mechanically assembling small molecules into a wide range of cheap, high-performance products at a very high rate (note that my definition of advanced nanotechnology is only loosely related to what people tend to mean by the term “nanotechnology”). (more)
  • If developed, advanced nanotechnology could increase existential risk, for example by making destructive capabilities widely accessible, by allowing the development of weapons that pose a higher existential risk, or by accelerating AI development; or it could decrease existential risk, for example by causing the world’s most destructive weapons to be replaced by weapons that pose a lower existential risk. (more)
  • Timelines for advanced nanotechnology are extremely uncertain and poorly characterised, but the chance it arrives by 2040 seems non-negligible (I’d guess 1-2%), even in the absence of transformative AI. (more)
  • It seems likely that there’d be a long period of development with clear warning signs before advanced nanotechnology is developed, pushing against prioritising work in this area and pushing towards a focus on monitoring and foundational work. (more)
  • There has been relatively little high-quality nanotechnology strategy work, and by default this seems unlikely to change much in the near future. (more)
  • It seems possible to make progress in this area, for example by clarifying timelines, tracking potential warning signs of accelerating progress, and doing strategic planning. (more)
  • Overall, I think that nanotechnology strategy research could be very valuable from a longtermist EA perspective. Currently, my extremely rough, unstable guess is that we should have 2-3 people spending at least 50% of their time on this by 3 years from now (against a background of perhaps 0-0.5 FTE over the past 5 years or so). (more)
  • Note that it seems that we don’t want to accelerate progress towards advanced nanotechnology because of (i) the dramatic but highly uncertain net effects of the technology, including the possibility of very bad outcomes, (ii) the plausible difficulty of reversing an increase in the rate of progress, and (iii) the option of waiting to gain more information. (Though note that I still feel a bit confused about how harmful various forms of accelerating progress might be, and I’d like to think more carefully about this topic.) (more)
Introduction

This post has two main goals:

  1. To provide a resource that EA community members can use to improve their understanding of advanced nanotechnology and nanotechnology strategy.
  2. To make a case for nanotechnology strategy research being valuable from a longtermist EA perspective in order to get more people to consider working on it.

If you’re mostly interested in the second point, feel free to quickly skim through the first parts of the post, or maybe to skip directly to How to prioritise nanotechnology strategy research.

Definitions and intuitions

In this section, I introduce some concepts that seem useful for thinking about nanotechnology strategy. I’ll refer to these in various places throughout the rest of the post.

Note that, although the focus of nanotechnology strategy research is ultimately advanced nanotechnology, I start by describing atomically precise manufacturing (APM), which I consider to be a particular version of advanced nanotechnology. I do this because APM is a far more concrete and well-explored technology than advanced nanotechnology, and because I like to refer to APM to help describe what I mean by advanced nanotechnology.

Defining atomically precise manufacturing (APM)

I think the term APM is often used in a fairly vague way, and doesn’t have a widely accepted precise definition, so for the purposes of this post I’ll introduce some more precisely defined concepts that pick out important aspects of what people refer to as APM.

These are:

  • Core APM: A system of nanoscale atomically precise machinery that can mechanically assemble small molecules into useful atomically precise products.
  • Complex APM: Core APM that is highly complex, made up of very stiff components, operates in a vacuum, and performs many operations per second; that is joined to progressively larger high-performance assembly systems in order to allow a range of product sizes; where the assembly machines and products are perhaps only mostly atomically precise; and where the assembly method is perhaps only mostly mechanical assembly.
  • Consequential APM: Complex APM that can create a wide range of complex, very high performance products at very low cost ($1000/kg or less) and with very high throughput (1kg of product per 1kg of assembly machinery in 3 days or less).

For more detailed definitions of these terms, see the Appendix section APM definitions in more detail.

The APM concept originated with Eric Drexler.[1] Complex APM roughly corresponds to the technical side of Drexler’s vision for APM, while consequential APM describes a watered-down version of the capabilities Drexler describes for APM.[2],[3] In what follows, I’ll use these terms when I want to be specific, and I’ll use the term “APM” when I want to point to the wider concept.

Intuitions for why APM might be possible and impactful

This section provides some quick intuitions for why you might think that consequential APM is possible and for why atomic precision might be desirable when building very high performance assembly machines and products.

We know that core APM is feasible, i.e. that atomically precise nanoscale machines can be used to do mechanical assembly of small molecules into useful atomically precise products, because examples of core APM exist in nature. For example, ribosomes are atomically precise nanoscale machines[4] that join amino acids to create proteins, which are themselves atomically precise.[5] (Note that we don’t have similar “existence proofs” for complex or consequential APM.)

Atomic precision might be desirable for nanoscale assembly systems and products because, on the nanoscale, the atomic building blocks might be 1/100th or 1/1000th the width of the structure, so that you need atomic precision unless you design for very wide tolerances in structural parts. Atomic precision also gives you perfectly faithful realisations of your design (provided there are no manufacturing errors).

A manufacturing capability that can exactly reproduce a target structure down to the last atom might intuitively seem able to produce products with performance dramatically exceeding current capabilities. Naturally evolved systems often outperform present-day artificial ones on important dimensions,[6] showing that we have some way to go before we reach performance ceilings. In addition, these evolved systems are composed of nanoscale machines and structures (albeit not always atomically precise ones), which are themselves a product of nanoscale manufacturing, suggesting that this is a powerful scheme for producing high performance products.

Nanoscale machines might be able to achieve high throughput because stiff nanoscale machines moving small distances can operate with very high frequency, and because 1cm³ of nanoscale assembly systems can together perform vastly more operations per second than a single machine of size 1cm³.[7],[8]

This high throughput in turn suggests cheap products, for the following reasons:

  • A complex APM system might be able to do efficient processing of input materials into whatever high-quality building blocks are required just as easily as it can do highly efficient assembly of those building blocks, and so be able to accept cheap and abundant input materials (only requiring the presence of the necessary chemical elements).
  • Similarly, we might expect efficient processing to allow for harmless, easily manageable waste.
  • A complex APM system might itself be cheap if it can manufacture copies of itself.

We can also appeal to naturally evolved systems, which seem to often be significantly cheaper to manufacture (at least based on a rough comparison of energy cost) than artificial ones, showing that nanoscale manufacturing systems can do cheap manufacturing.[9]

For a more detailed discussion, see Appendix section More intuitions for why APM might be possible and impactful.

Broadening the scope from APM to advanced nanotechnology

As far as I know, EA efforts in this area have been focused on APM. I consider APM to be a remarkably concrete vision for future nanotechnology, and an extremely useful thing to analyse, but I tentatively propose that we slightly broaden the scope of work in this area to consider what I call advanced nanotechnology.

I define advanced nanotechnology as any highly advanced technology involving nanoscale machinery that allows us to finely image and control processes at the nanoscale, with manufacturing capabilities roughly matching, or exceeding, those of consequential APM.

Advanced nanotechnology covers a wider range of possible future technologies than APM. For example, a future nanotechnology might rely less on mechanical positioning or very stiff machines than APM does.[10],[11] But these technologies would look similar to APM in many ways, and might have the same strategic implications. My current feeling is that it’s better for strategy work to cover this wider area than to focus purely on APM.[12]

Advanced nanotechnology vs nanotechnology today

Note that advanced nanotechnology is fairly loosely connected to the concept that’s usually pointed to by the term “nanotechnology”.

The term “nanotechnology field” seems to commonly refer to a loose connection of research areas that are united by the fact that they concern physical systems on a length scale of between roughly 1-100nm (although there are lots of fields that aren’t usually considered nanotechnology that also concern physical systems on that length scale). The term “nanotechnology” is sometimes used to refer to useful (potential) products or technologies that come from this field.

See the later section called Current R&D landscape for my take on the present-day fields of research that are most relevant for advanced nanotechnology.

Defining transformative AI (TAI)

At a few points in this post I’ll refer to “transformative AI”, usually through the abbreviation “TAI”. By this I mean “AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution”, which is the definition for TAI sometimes used by Open Philanthropy (for more detail on this definition, see What Open Philanthropy means by "transformative AI").

Potential effects of advanced nanotechnology

Advanced nanotechnology’s extremely powerful capabilities suggest that its development could have very dramatic effects.

The potential effects listed below seem among the most important from a longtermist EA perspective. I’d guess that if advanced nanotechnology is developed within the next 20 years and we haven’t (yet) developed TAI, there’d be something like a 70% chance that it would have very dramatic effects, i.e. effects of at least similar importance to the ones described below.[13] Note that the first two bullet points use the typology from Nick Bostrom’s paper The Vulnerable World Hypothesis.

  • Vulnerable world type-1 vulnerabilities. Advanced nanotechnology might lead to widespread access to manufacturing devices able to make things like nuclear weapons, dangerous pathogens, or worse. This seems like a plausible outcome to me, assuming society were to fail to properly regulate and control the use of the technology.
  • Vulnerable world type-2a vulnerabilities. Even if manufacture of highly destructive weapons doesn’t become widely available to ordinary citizens, advanced nanotechnology might allow states to manufacture weapons that pose an (even) greater probability of existential catastrophe than the highly risky weapons that are currently accessible (such as currently accessible nuclear and biological weapons). I’d place a “grey goo” scenario, where self-replicating nanoscale machines turn the entire world into copies of themselves, into this category. It seems very plausible to me that advanced nanotechnology would enable the development of weapons that pose a significantly larger existential risk than do our most dangerous present-day weapons. And while weapons that have a moderate-to-high chance of causing existential catastrophe might not seem very appealing on the face of it, it seems very plausible to me that at least some states would perceive a sufficiently high strategic benefit from developing them.[14]
  • Reduced existential risk from state weapons. On the other hand, in a reversal of the above scenario, advanced nanotechnology might lower existential risk from nuclear and biological weapons (or from other future weapons that pose an existential risk) by allowing states to develop weapons that are more strategically useful and pose a lower existential risk. This also seems very plausible to me.
  • More powerful computers, leading to earlier TAI. Advanced nanotechnology could allow us to build more powerful computers more cheaply. If advanced nanotechnology is developed before TAI, having cheaper and better computers could lead to earlier TAI.[15] This seems very plausible to me, and I’d also (very tentatively) guess that this would come earlier than the effects mentioned in the previous three bullet points.

Each of the above effects could constitute an existential risk or existential risk factor, or could correspond to a reduction in existential risk. Overall, currently I’m very unsure about whether advanced nanotechnology would be good or bad for the world on net, and I’m very unsure whether I’d rather learn that advanced nanotechnology was coming sooner or later than expected (but I still currently think that acting to speed up the arrival of advanced nanotechnology is probably bad. See the later section Potential harms from nanotechnology strategy research).

Note that the potential effects described above all follow from having an extremely powerful and flexible manufacturing capability, and novel nanoscale devices as products are necessary only in the grey goo scenario.[16]

For other potential effects of advanced nanotechnology, see the Appendix section Other potential effects of advanced nanotechnology.

When and how advanced nanotechnology might be developed

This section is quite long and complex, so here is a quick summary:

  • My rough guess is that there’s something like a 20-70% chance that advanced nanotechnology is feasible (where by “feasible” I mean: it’s possible in principle for a sufficiently advanced civilisation to build the technology, ignoring issues of economic feasibility). (more)
  • I’d guess there’s currently something like $10-100 million per year of funding that is fairly well directed towards developing advanced nanotechnology. These efforts are mostly focused on using scanning probe microscopy, which doesn’t seem like the most effective approach for developing advanced nanotechnology. (more)
  • My current very rough, unstable guesses regarding advanced nanotechnology timelines are:
    • Assuming feasibility, a median estimate of 2110 for the arrival of advanced nanotechnology. (more)
    • Not assuming feasibility, a 4-5% probability that advanced nanotechnology arrives by 2040. (more)
    • Not assuming feasibility, and assuming that advanced nanotechnology comes before TAI, a 1-2% probability that advanced nanotechnology arrives by 2040. (more)
  • It seems likely to me that there’d be a long period of development with clear warning signs before the arrival of advanced nanotechnology. (more)
In-principle feasibility

In this section, I briefly give my views on whether APM technologies, and advanced nanotechnology more broadly, are feasible in principle. By “feasible in principle”, I mean that it’s possible in principle for a sufficiently advanced civilisation to build the technology, ignoring issues of economic viability.

The probabilities given in this section are based on only relatively shallow thinking and are very unstable, and shouldn’t be taken as more than a rough indication of my personal guesses at the time I wrote this.[17]

As noted in the earlier section Intuitions for why APM might be possible and impactful, we know that core APM is feasible because examples of it exist in nature.

Maybe I’d give a 50% probability that complex APM is feasible. While I think there’s a significant chance that complex APM is feasible, and I’m not aware of convincing arguments that it couldn’t possibly be built, I wouldn’t find it that surprising if it turns out to be impossible. For example, it might turn out that it’s not possible to find suitable arrangements of atomic building blocks, given the finite selection of atoms and atomic sizes.

Assuming that complex APM is feasible, it seems pretty plausible to me that consequential APM is also feasible. Maybe I’d give this a 10-50% probability (implying that I guess a 5-25% unconditional probability that consequential APM is feasible). The earlier section Intuitions for why APM might be possible and impactful and the Appendix section More intuitions for why APM might be possible and impactful cover some reasons for thinking that complex APM might be feasible. In addition to those points, I’d note that Drexler has published technical arguments for the in-principle feasibility of APM (including a book, Nanosystems, which explores designs and capabilities), and in my view no-one has shown that the technology is not feasible despite some attention on the question.[18]

I don’t assign a higher probability to the feasibility of consequential APM because I feel like there could be practical barriers that block the development of consequential APM even if complex APM is feasible. For example, maybe the system can’t be made reliable enough or is very expensive to maintain (so that the “very cheap products” condition can’t be met), or maybe there just aren’t any designs that allow you to take advantage of the favourable theoretical properties of nanoscale assembly systems.

Advanced nanotechnology is, by definition, at least as likely to be feasible as consequential APM, and we might consider it to be substantially more likely to be feasible because it covers a much wider array of potential technologies. Overall, I’d guess something like a 20-70% chance that advanced nanotechnology is feasible. This is around 3 times as large as my guess for the probability that consequential APM is feasible, which feels reasonable to me.

Current R&D landscape

Anecdotally, my impression is that the majority of researchers in fields relevant for advanced nanotechnology haven’t heard of APM, and don’t think about visions for transformative nanotechnology more generally.[19]

Correspondingly, there is not much R&D explicitly targeting APM or broader advanced nanotechnology, as far as I’m aware.

I’d guess there’s something like $10-100 million per year of funding that is fairly well directed towards developing advanced nanotechnology. This is mostly being spent (according to my guess) at Zyvex and on a project at Canadian Bank Note. These projects use an approach involving scanning probe microscopy[20] to make progress towards APM (sometimes called the “hard path” to APM), which doesn’t seem like the most promising approach,[21] although of course surprises are always possible.

Particular examples of less well-targeted, but still relevant, work include:

More broadly, my impression is that the most relevant work for progress towards advanced nanotechnology is:

  • Work in the protein engineering and DNA nanotechnology fields.
  • Work on spiroligomers and foldamers, as well as supporting technologies such as computational approaches (AlphaFold is a notable example relevant for protein engineering) and nanoscale imaging.
  • Work on advancing scanning probe microscopy.
  • Perhaps to a lesser extent, a broad class of work on non-biological dynamic nanoscale systems.[22]
Timelines Considerations relevant for thinking about APM timelines

In this section, I’ll discuss considerations relevant for thinking about when APM might be developed. I focus on APM here rather than broader advanced nanotechnology because I find it easiest to first think about APM and then consider how much earlier things might happen if we consider timelines to advanced nanotechnology of any form, rather than just timelines to APM.

The most obvious consideration pushing against short timelines for APM is the very slow rate of progress towards APM in the last 35 years. For example, atomic manipulation with scanning probe microscopes doesn’t seem to have improved very much since the original demonstration of the technique by IBM in 1989. In addition, despite significant progress in protein engineering, positional assembly with protein suprastructures (which is perhaps the first step on a hypothesised pathway to APM called the “soft path”, as discussed in footnote 5) has still not been demonstrated.[23] Overall, my very rough (and unstable) guess would be that we’ve come perhaps 10% of the way to APM in the past 35 years.

Aside from the empirical observation of slow progress, my inside-view impression is that it would be an enormous engineering challenge to make progress along the technical pathways sketched for APM. Further down the line, it also seems hugely challenging to engineer complex APM systems; although perhaps this is mitigated by the consideration that very stiff, later-stage systems might be much more predictable (and so easier to model and engineer) than early-stage ones.

An additional consideration pushing against short timelines is that it doesn’t seem like there would be many commercial applications from making the first few steps towards APM (even though later stages might have lots of commercial applications). This reduces the incentive for private research efforts and reduces researcher interest within academia.

Some considerations push in favour of shorter timelines for APM, however.

Firstly, in outlining the “soft path” to APM (see footnote 5), Drexler has sketched a technical pathway to APM that I find plausible.

In addition, it seems plausible that an APM field could emerge over the next decade, or perhaps that some highly targeted and well-resourced private effort will emerge. Because there has been relatively little highly targeted effort towards the development of APM so far, we have relatively little empirical evidence regarding what rate of progress would be possible under such an effort; perhaps progress would be much faster than expected.[24]

Further, advances in AI are leading to advances in our ability to model molecular systems, most notably in the field of protein folding, which is particularly relevant for near-term progress along the soft path to APM. It seems plausible to me that increasingly powerful AI, even if it falls short of TAI, will lead to surprisingly fast progress towards APM.

Finally, it seems very possible that the arrival of TAI would lead to the rapid development of APM. This might significantly shorten your timelines for APM, depending on your timelines for TAI.

Median timelines

As mentioned above, my guess for the probability that advanced nanotechnology is feasible in principle fluctuates between roughly 20% and 70%. If my probability for feasibility is less than 50%, it doesn’t really make sense to talk about a median timeline: if I think the chance that it’s possible to develop advanced nanotechnology is less than 50%, it wouldn’t make sense to think that at some point in the future there’s a 50% chance that we’ve developed advanced nanotechnology.

Assuming advanced nanotechnology is feasible in principle, my median timeline is perhaps something like 2110. But this number is very made up and not stable. It also interacts pretty strongly with my timelines for TAI, since TAI could massively accelerate technological progress.

Another data point comes from Robin Hanson in 2013, when he reported a guess of roughly 100-300 years until something seemingly roughly equivalent to advanced nanotechnology is developed.

Probability of advanced nanotechnology by 2040

We might be particularly interested in the probability that advanced nanotechnology arrives in the next couple of decades, since events further in the future seem generally harder to influence.

We might also be particularly interested in worlds where advanced nanotechnology is not preceded by TAI, because, for example, we might think that after TAI “everything goes crazy” and it’s hard to make useful plans for things that happen after that point. Personally, I think it’s reasonable to imagine that work on nanotechnology strategy will be much less useful if advanced nanotechnology is preceded by TAI, although I’m not at all confident about this and my view feels quite unstable.

I’d guess there’s something like a 1-2% chance that advanced nanotechnology arrives by 2040 and isn’t preceded by TAI.[25] As with my median timeline, this number is very made up and not stable.[26]

For the probability that advanced nanotechnology arrives by 2040 and is preceded by TAI, my current speculative guess is something like:

“16% chance of TAI by 2040”
×.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} “50% chance that advanced nanotechnology is feasible in principle”
× “40% chance that advanced nanotechnology is developed between TAI and 2040, given that TAI arrives before 2040 and that advanced nanotechnology is feasible in principle”
= 3% chance that advanced nanotechnology arrives by 2040 and is preceded by TAI.

You might want to substitute your own probabilities for the arrival of TAI and/or the chance that TAI leads to the development of advanced nanotechnology.

Overall, then, adding the above probabilities implies that my guess is that there’s a 4-5% chance that advanced nanotechnology arrives by 2040. Again, this number is very made up and not stable.

Rate of progress and warning signs

I imagine that developing advanced nanotechnology would be a huge engineering challenge, and there would be a lengthy road to get there from where we are today. In the median case, I imagine that it would be extremely obvious for many years that this kind of work is being done: the field might look a bit like the present AI field, for example, with notable private-sector efforts alongside lots of work in academia. It seems very unlikely, though possible, that progress would be driven by some huge, secretive, Manhattan Project–style effort instead; but even in that case, it seems like informed people would probably know that something was up, for example by noticing that relevant academics had suddenly stopped publishing.

The main ways I can see the above picture being wrong are:

  • Technical progress just turns out to be way easier than it seems now. Maybe surprisingly quick progress along the “hard path” to APM (i.e. making progress using scanning probe microscopy) is the most likely way this could happen.
  • Pre-transformative AI dramatically speeds up the rate of progress in relevant fields.
  • Transformative AI dramatically speeds up technological progress (but then maybe “all bets are off” anyway).

It’s worth noting that if we’re focused on worlds where advanced nanotechnology arrives in the next 20 years, we should presumably focus more than we would otherwise on worlds where progress happens surprisingly quickly. This is because worlds where advanced nanotechnology arrives in the next 20 years will tend to be worlds where progress happens surprisingly quickly (for example, if there is a gradual ramp up in progress over the next 30 years, we obviously won’t have the technology in 20 years’ time).

Current landscape for nanotechnology strategy

I’m aware of little thoughtful nanotechnology strategy work currently being undertaken, and I don’t think there’s a very large amount of high-quality existing work.

The most notable public EA work in this area that I’m aware of is a 2015 Open Philanthropy report called Risks from Atomically Precise Manufacturing.

As far as I know, no-one in the EA community, other than me, is currently spending a significant fraction of their time thinking about this or is planning to do so. Due to other (potential) projects competing for my time, I’d guess I’ll spend something like 25% of my time on nanotechnology strategy-related things on average in the next 12 months, with a good chance (maybe 50%) that I don’t spend much time at all on nanotechnology strategy things in the next 12 months. Maybe I’d guess I’ll spend 40% of my time on average on nanotechnology strategy things over the next 5 years.

Outside of EA, the Foresight Institute has historically thought about nanotechnology strategy questions, and they have perhaps 0.2-0.6 FTE working in this broad area currently. Other organisations have been active in this area, such as the Center for Responsible Nanotechnology and the Institute for Molecular Manufacturing,[27] but I’m not aware of relevant recent work. I think that a lot of the public work in this area has been of low quality, and overall I think the existing work falls a long way short of providing complete and high-quality coverage of the important questions within this area.

Prospects for making progress

The high-level goals of EA work in this area could be to reduce the chance of an existential catastrophe due to advanced nanotechnology, to use its development to reduce existential risk from other sources, or more broadly to achieve any attainable value related to its development.

Potentially valuable interventions might include:

  • Seeking to speed up particular technical research areas where that seems to promote safety.
  • Making policy recommendations related to regulating, monitoring, and understanding emerging advanced nanotechnology.
  • Positively steering the development of advanced nanotechnology by being a major early investor or founder.

We have some reasons to expect progress to be hard in this area: the technology appears likely to be many decades away, and the terrain for strategy work is poorly mapped out at present.

On the other hand, the fact that the world is not paying much attention to advanced nanotechnology gives altruistic actors a chance to make preparations now in order to, for example, be in a position to make impactful policy recommendations at critical moments later. In addition, the relative lack of exploration makes it hard to rule out the existence of valuable low-hanging fruit. And, regarding our understanding of the technology itself, advanced nanotechnology is more opaque than risky biotechnology, but perhaps compares favourably with artificial general intelligence (AGI), since APM is arguably a more plausible blueprint for advanced nanotechnology than present-day machine learning methods are for AGI.

Examples of concrete projects

One concrete project could be to consider what you would do if you knew that advanced nanotechnology was coming in 2032, seeking areas of consensus among EA-aligned individuals who are well informed about nanotechnology strategy. The results could inform what we should do now (given our actual timelines), perhaps pointing to a particular high-value intervention. The results could also help generate a plan to hold in reserve in case our advanced nanotechnology timelines were to dramatically shorten, increasing the chance that EAs successfully execute high-value interventions in the case of shortened timelines. In addition, the exercise would be practice for future strategic thinking, potentially increasing the chance that EAs successfully execute high-value interventions at some future time after a change of circumstances or further deliberation.

Another project could be to clarify what the key effects of complex APM that we want to forecast are; work out the technical requirements of these effects; and create forecasts using trend extrapolation, other outside-view considerations and methods, and inside-view judgements. These forecasts could help prioritise nanotechnology strategy research against work in other cause areas. That way, if advanced nanotechnology timelines turn out to be shorter than initially believed, more effort can be exerted in this area, increasing the chance that EAs and others successfully identify and execute high-value interventions. Forecasts of various key effects could also be helpful for directing work within nanotechnology strategy towards the most high-value sub-areas, perhaps increasing the chance that high-value interventions are found and successfully executed.

A third potential project could be to identify and monitor potential warning signs of surprising progress towards advanced nanotechnology, for example by identifying the most relevant areas of current research, the most important research groups, and key bottlenecks and potential breakthroughs. Similarly to the previous project, this monitoring could help prioritise nanotechnology strategy research against work in other cause areas, so that more resources are expended in this area if progress appears to be accelerating, increasing the chance that EAs and others successfully identify and execute high-value interventions.

How to prioritise nanotechnology strategy research

This section gives some arguments and considerations relevant for prioritising nanotechnology strategy research against other cause areas from a longtermist EA perspective. To skip to my personal bottom line view on this, see the final subsection of this section, My view on how longtermist EAs should be prioritising this work against other areas right now.

A case for nanotechnology strategy research

Pulling together the information presented in the previous sections, a rough, high-level argument for nanotechnology strategy research being highly valuable from a longtermist EA perspective could be the following:

  1. Advanced nanotechnology could have dramatic effects, with both positive and negative potential implications for existential risk.
  2. There seems to be a non-negligible (although low) probability that it will arrive within the next 20 years, even if it’s not preceded by TAI (my wild guess is that there’s a 1-2% probability of this).
  3. There’s been very little high-quality work in this area.
  4. It seems possible to make progress in this area.[28],[29] I’d also note that I think there’s a synergy between nanotechnology strategy and other areas like AI governance and biosecurity, which makes work in nanotechnology strategy more tractable and more valuable than it would be otherwise. Specifically, because these areas all concern trying to make the development of very powerful emerging technologies go well for the world, methodologies and findings seem likely to be transferable between these areas to some extent.
Could we wait until advanced nanotechnology is about to be developed?

As discussed in the earlier section Rate of progress and warning signs, my current guess is that there’s very likely to be a long, obvious-to-the-outside-world R&D process before advanced nanotechnology is developed. This pushes in favour of deprioritising this area until there are signs that progress towards advanced nanotechnology is speeding up.

However, this depends on to what extent nanotechnology strategy work needs to be done sequentially versus in parallel. The more the work needs to happen sequentially, the more it’s worth doing work now. I feel very uncertain about this, although it seems like the work is probably at least a bit sequential. For example, maybe it takes time to lay the conceptual foundations for a new topic, and also presumably it takes time for people to get up to speed and start making contributions.

In addition, we might think that being one of the first to act in this space when progress starts to pick up is valuable. I’d guess there could be a fair amount of value here.

Overall, I think the likely slow and easy-to-detect ramp up towards advanced nanotechnology pushes in favour of spending relatively few resources now to do foundational work, while carefully monitoring progress and being ready to pivot to spending lots of EA effort in the area if that seems important.

Potential harms from nanotechnology strategy research

Nanotechnology strategy is a very complex area, and there are many potential harms from well-intentioned nanotechnology strategy work.

It currently seems to me that we don’t want to speed up progress towards advanced nanotechnology. Roughly speaking, I think this because i) the potential effects of the technology seem dramatic, but very uncertain (they could be very good or very bad), ii) it seems plausible that accelerating progress will be hard to reverse, and iii) we have the option of waiting and potentially gaining more information about the effects of the technology. It seems plausible that accelerating progress will be hard to reverse because, for example, visibly faster progress could generate more interest and a stronger research effort, which in turn sustains the rate of progress. (Though note that I still feel a bit confused about how harmful various forms of accelerating progress might be, including whether some forms would be harmful at all, and I’d like to think more carefully about this topic.)

Because accelerating progress seems bad, public talk about advanced nanotechnology in a way that hypes the technology or otherwise generates interest in developing it could cause harm by speeding up progress towards the technology. And it seems relatively easy to notably increase attention on the technology given that there’s currently relatively little attention on it.[30] Emphasising military applications seems particularly undesirable, since it might tend to push the direction of the technology’s development towards dangerous applications.

In addition, findings from nanotechnology strategy research could, in some cases, represent information hazards that would inform efforts to speed up advanced nanotechnology development. Examples include findings from efforts to better understand advanced nanotechnology timelines by mapping out technical pathways, or from mapping out scenarios for the development of an advanced nanotechnology field. While it’s possible in principle to keep these findings private, there are advantages to public communication, and exchanging ideas with people interested in speeding up the development of advanced nanotechnology is often very helpful for doing nanotechnology strategy work. So the right policy here isn’t always obvious.

Because this is a relatively unexplored area, maybe there’s also a risk that poor initial attempts at nanotechnology strategy research will “poison the well”, doing long-term damage to the ability for progress to be made in this area. This could happen because researchers develop a poor set of concepts, or because EA-aligned people communicate with policymakers in a poorly thought-out way that discourages them from engaging with EA-aligned people in the future.

There may also be a risk that nanotechnology strategy research leads to making policy recommendations that turn out to be harmful and hard to reverse. For example, this could occur if a policy recommendation seems beneficial after a shallow investigation, but enough thought would show it to be harmful.

Relatedly, some individual or group of nanotechnology strategy researchers might mistakenly come to believe that accelerating progress towards advanced nanotechnology is a worthwhile goal, and then act to accelerate progress, causing irreversible damage. (To be clear, I also think it’s possible that at some point we’ll correctly determine that accelerating advanced nanotechnology progress is a worthwhile goal.)

Mitigating the risks of harm

To reduce the risk from these potential harms, prospective nanotechnology strategy researchers require good judgement and a strong support network they can turn to for high-quality advice and feedback. Keeping the unilaterist’s curse in mind also seems important.

Prospective funders in this area should keep these traits in mind when considering who to fund. In addition, funders could, for example, look for an assurance that nanotechnology strategy researchers won’t share their work outside a trusted circle without particular trusted individuals giving approval.

Why you might not think work in this area is high priority

Here are a couple of reasons you might not consider work in this area to be high priority from a longtermist EA perspective.

Firstly, you might think that it’s just too difficult to make progress towards concrete impact, because advanced nanotechnology is likely to be far off in time, and hard to analyse because its precise nature is very uncertain. Tangible interventions are notably absent right now.

I think this is a reasonable position, although I disagree with it. I think there are concrete projects that seem useful (see the earlier section Examples of concrete projects), I think that this area is too unexplored to be very sure that there isn’t valuable work here, and I think that in scenarios where advanced nanotechnology arrives in the next couple of decades we’ll be glad that work was done in the early 2020s.

Secondly, you might just be convinced that some other area is much more important from a longtermist EA perspective, so that you don’t think longtermist EA resources should be spent on nanotechnology strategy research. For example, maybe you have short AI timelines and think AI safety work is at least moderately tractable.

Again, I think this position is reasonable (either regarding the great importance of AI safety work, or some other area), although it’s not a position I hold — cause prioritisation is hard, and there’s lots of scope for reasonable people to disagree.

My view on how longtermist EAs should be prioritising this work against other areas right now

In this section I’ll briefly give some more concrete views on how people from the longtermist EA community should prioritise working on this compared to working in other areas. I haven’t thought about this a huge amount (and this kind of prioritisation is extremely difficult and also strongly depends on details that I’m brushing over), so these are extremely rough views and are liable to change in the near future. But reviewers of earlier drafts of this post were keen to see something like this, so I’m providing it here.

I’d guess that some small, non-zero fraction of EA resources should be going into nanotechnology strategy research right now. In particular, a rough guess might be that we want to move to having 2-3 people each spending at least 50% of their time on this in the next 3 years, and maybe get to 4 FTE in 5 years.

So I don’t think we should be piling a huge amount of resources into this, but the resource allocation I’m suggesting comes against a background of (it seems to me) something like 0-0.5 FTE of longtermist EAs thinking about this at any given time over the past 5 years. So this would represent a significant change from the status quo.

Another angle on this is my view on the following highly abstract case: if I had an aspiring researcher in front of me who seemed about an equally good fit for nanotechnology strategy research, biorisk research, AI safety research, and AI governance research, I’d rank nanotechnology strategy research a bit below the others. I’d rank the options in this way because I think we should be able to recruit enough people from the pool of people who are a better fit for nanotechnology strategy research than for research in other areas (but again, I feel very uncertain about all of this).

Some guesses at who might be a good fit for nanotechnology strategy research

Being broadly longtermist EA-aligned seems important for doing work in this area.

In addition, I’d guess that, very roughly speaking, the following sorts of people would be a particularly good fit for nanotechnology strategy research:

  • People with chemistry/physics/biology/materials science backgrounds (among others), and especially people with PhDs in those areas.
  • People who have done this kind of “strategy” thinking in other contexts, including in other EA areas like AI risk or ending factory farming.
  • People who are thoughtful, have good judgement, and are not likely to act unilaterally.
  • People who are willing to build the support network mentioned in Mitigating the risks of harm.
  • People willing to tackle something hard and relatively unexplored, and willing to go for higher-risk things on the basis of high expected impact.
  • People who are okay with doing work that might be less legible to the outside world because of infohazard concerns and the unpredictable nature of exploratory research.
Conclusion

Advanced nanotechnology could significantly increase or decrease existential risk, and might arrive in the next couple of decades, even without transformative AI. So far there’s been relatively little high-quality work aimed at making its development go well for the world. Doing nanotechnology strategy research now can help lay the foundations to increase the chance that the development of the technology goes well.

I would love to see more people considering working in this area. If you’re interested in learning more, and especially if you’re interested in trying out work in this area now or later in your career, please get in touch by sending me a private message on the Forum.

Acknowledgements

This post was written by me, Ben Snodin, in my capacity as a researcher at Rethink Priorities. I finished writing this post while at Rethink Priorities, and I did a lot of the work for it prior to joining Rethink Priorities, while employed as a Senior Research Scholar at the Future of Humanity Institute. Many thanks to James Wagstaff, Jennifer Lin, Ondrej Bajgar, Max Daniel, Daniel Eth, Ashwin Acharya, Lukas Finnveden, Aleš Flídr, Carl Shulman, Linch Zhang, Michael Aird, Jason Schukraft, and others for their helpful feedback, and to Katy Moore for copy editing. If you like Rethink Priorities’ work, you could consider subscribing to our newsletter. You can see more of our work here.

Appendix Further reading

Some resources that are especially relevant for nanotechnology strategy research are:

These resources are also useful for context and technical understanding:

APM definitions in more detail

This section provides more detailed definitions for the core APM, complex APM, and consequential APM concepts introduced in the main text in Definitions and intuitions.

I define core APM as:

A system of nanoscale, atomically precise machines that can mechanically assemble small molecules into useful atomically precise products.

Where:

  • By nanoscale I mean “extremely small”, roughly of length 1-100nm in each dimension. For comparison, atoms often have a diameter of around 0.1-0.2nm. Note that a core APM system, while composed of nanoscale machines, might itself be larger than 100nm.
  • Atomically precise machines means machinery that has exactly the desired atomic structure. Similarly, atomically precise products means products that have exactly the desired atomic structure.
  • Mechanically assemble small molecules means (in this context): use nanoscale machines to control the motion of small molecules, using short-range intermolecular forces, such that the molecules form stable bonds with some partially constructed product.

We can describe a particular form of core APM that uses complex, very stiff components and performs many operations per second. In addition, we can imagine that the products from the nanoscale assembly system are used as inputs to slightly bigger (but otherwise very similar) assembly systems, which in turn assemble them into bigger products; and so on until you get to whatever size product you want (including everyday-scale things like laptops). Finally, we can slightly relax the definition to include systems where the assembly machines and products are only mostly atomically precise, and where the assembly method is only mostly mechanical assembly.

I label such a technology complex APM (the modifications to the core APM concept are in bold):

An extremely complex, intricate system of very stiff nanoscale, (mostly) atomically precise machines joined to progressively bigger assembly systems that operate in a vacuum and perform many operations per second to (mostly) mechanically assemble small molecules into (mostly) atomically precise products with high throughput, and with product sizes ranging from from nanoscale to metre-scale and beyond.

Finally, Drexler claims that these APM systems could create a wide range of complex, very high performance products very cheaply and with very high throughput. In this spirit, I define consequential APM to refer to:

A complex APM technology that can create a range of complex, very high performance products for $1000/kg or less and with a throughput of 1kg every 3 days or less per kg of machinery.

More intuitions for why APM might be possible and impactful

This section expands on the Intuitions for why APM might be possible and impactful described in the main text. As in the main text, these arguments are designed to give the reader some feeling for why you might believe these things rather than constituting cast-iron proofs.

See Drexler’s 1992 book, Nanosystems, for a treatment of the physics underlying nanoscale manufacturing systems. Chapter 2 covers the scaling of important physical properties with system size and is particularly relevant for intuitions about nanoscale manufacturing technology.

Intuitions for why atomic precision might be a natural goal

If you want to build high-performance systems and products on the nanoscale, it’s perhaps natural to build systems and products with atomic precision.

As mentioned in the main text, if you have small molecules as building blocks and you’re making products on the order of 1-100nm, you’re building things out of discrete building blocks with a width roughly 1/10th to 1/1000th of the width of the product you’re building. You may well then need atomic precision in order to build the product to the correct specification.

In addition, molecular building blocks are perfect copies (e.g. all oxygen molecules are the same[31]), which bond together in discrete ways to form a finite set of possible structures. So by insisting on atomic precision, you get perfectly faithful physical realisations of your design.[32]

Finally, given high-performance systems that are able to manufacture things with an extremely high degree of control, and if atomically precise products will often give significantly higher performance, atomically precise products will perhaps be an attractive target.

Intuitions for high-performance products

One reason we might expect complex APM to produce a very wide range of complex products is that building things by successively adding small molecules would seem to allow for huge flexibility in products. We see something similar with 3D printers today, which successively add material to a growing structure and can create a wide range of complex structures.

To give some intuition for the claim that atomically precise products can achieve very high performance, the semiconductor industry spends tens of billions of dollars annually on R&D to manufacture chips with increasingly finer-grained features, a pathway that ends with atomic precision (or close to it). (To be clear, I’d imagine that far more impressive products would be possible with flexible atomically precise fabrication.)

As mentioned in the main text, another angle is to consider that, with today’s technology, naturally evolved systems and devices often outperform present-day artificial ones on important dimensions. These evolved systems are certainly not atomically precise (although they contain some atomically precise components), but they show that we are currently some way from the maximum achievable performance along many dimensions. In addition, some people, including Drexler, have argued that evolved systems necessarily look different to designed ones, and that this doesn’t imply that designed systems must be inferior.[33] A highly flexible manufacturing capability that can exactly produce a vast array of complex products defined down to the last atom might seem able to meet and perhaps dramatically exceed the performance of evolved systems.

Intuitions for high operating speeds and high throughput

As mentioned in the main text, one reason to think that nanoscale machines might achieve high throughput is that these machines can in principle each operate very rapidly, because they are made of very stiff materials and each operation only needs to involve movements over very short distances.

As was also noted in the main text, another reason is that the machines are not too much bigger than the small molecules they are manipulating during the assembly process. This makes a high rate of production per gram of manufacturing system more plausible than in the case of fabrication using present-day atomic force microscopes, for example, where a machine weighing 0.1 grams (or more) performs assembly operations on single atoms weighing around 10-23 grams.

In addition to these considerations, the performance of biological machines suggests that complex APM could produce products fast. Some bacterial cells can double in both cell count and total mass of cells roughly every 10 minutes. Cells are not themselves atomically precise, but they include nanoscale atomically precise components, so we can consider this as an example of atomically precise components producing their own mass in product (copies of themselves) in 10 minutes, albeit as part of a larger system that is acting to produce a copy of all of its components.[34],[35]

Intuitions for low cost

As argued in the main text, high throughput could lead to cheap products from complex APM if the machinery uses cheap inputs; is itself cheap; and produces harmless, easily manageable waste.

We might expect the inputs to be cheap if they consist of readily available raw materials, with minimal processing required before they are passed to the APM system.

This might make sense if complex APM machinery is able to deal with minimally processed inputs, mostly requiring only that the right chemical elements are present in the inputs (not necessarily in the right ratios). This could be the case if the machinery was able to cheaply and efficiently do most of the required processing itself, which might seem plausible for complex APM machinery that is able to efficiently produce a wide range of atomically precise products through mechanical assembly.

Inputs containing the right chemical elements might be cheap if only commonly occurring elements are needed. One reason to think that only common elements are needed is that carbon, a relatively abundant element, seems to be an excellent building material: carbon forms diverse atomic-scale structures and has excellent physical properties (for example, it can exist as diamond, graphite, or carbon nanotubes, and diamond is extremely hard and stiff).

The complex APM machinery could be cheap because, once you know how to make it, perhaps you could use it to make lots more machinery, just as you could use it to make any other product.

We might expect the complex APM machinery to produce harmless, easily manageable waste because, as with processing inputs, we might expect that if we can build machinery that is able to efficiently produce a wide range of atomically precise products through mechanical assembly, we can also build similar machinery that can efficiently process waste into a harmless and easily manageable form.

Finally, as mentioned in the main text, we can also consider that naturally evolved systems seem to often be significantly cheaper to manufacture than artificial ones, at least by a rough comparison of energy cost according to the quick investigation by Paul Christiano described in Simple evolution analogy.

Other potential effects of advanced nanotechnology

Other than the potential effects mentioned in the main text in the section Potential effects of advanced nanotechnology, some (perhaps) less important effects of advanced nanotechnology being developed might be:

  • An “extremely powerful surveillance and non-lethal control” scenario. Advanced nanotechnology might allow states to cheaply manufacture ubiquitous, centrally controlled drones the size of insects or small mammals to monitor and control the world’s population using non-lethal force.[36] This seems like a plausible outcome to me. This capability might lead to or facilitate value lock-in (probably a very bad outcome), but it also seems possible that it would reduce existential risk by mitigating “vulnerable world type-1” vulnerabilities.
  • Advanced nanotechnology as a TAI capability. TAI could lead to the rapid development of advanced nanotechnology. This could allow an agent-like AI to quickly create more powerful hardware and thereby gain the ability to rapidly transform the physical world.
  • Impacts on biorisk. Advanced nanotechnology, or technologies along the path to it, seem likely to make it easier to develop things that could increase or decrease biorisk, including deadly pathogens, DNA sequencing devices, and biosensing devices. It might also enable a highly robust, rapid, local capability to manufacture medical countermeasures in a scenario where a global biological catastrophe has destroyed supply chains and important infrastructure.[37]
  • Advanced neurotechnology. Advanced nanotechnology might allow us to monitor the brain at very high resolution, potentially enabling whole brain emulation or neuromorphic AI. It seems unclear whether this would be good or bad overall.
  • Cheap energy. Cheap, powerful manufacturing might enable the fabrication of cheap solar cells and cheap batteries that help to overcome intermittency in solar power, leading to very cheap solar power (although, naively, I’m unsure how large an effect this would be given that advanced nanotechnology wouldn’t on the face of it reduce land and labour costs).
  • Reduced risks from catastrophic climate change. With advanced nanotechnology, we might be able to return atmospheric CO2 concentrations to pre-industrial levels for a relatively small cost using cheap, high-performance direct air capture devices powered by cheap solar. This could reduce the risk of catastrophic climate change, and also reduce existential risk if we believe that climate change poses such a risk.
  • Improved resilience to global catastrophes. As mentioned above, advanced nanotechnology might provide a highly robust and powerful local capability to manufacture medical countermeasures during a global biological catastrophe. More generally, the technology could enable greater resilience to global catastrophes by broadly enabling local, rapid production of vital goods, equipment, and infrastructure without requiring pre-existing infrastructure or supply chains, and regardless of environmental damage. One example might be food production without relying on sunlight or global supply chains (see also resilient foods). This could reduce the chance that a global catastrophe precipitates an existential catastrophe.
  • Dramatic improvements in medicine. Aside from an expectation that cheap, powerful manufacturing would be useful for fabricating medical products (just as for most other products), advanced nanotechnology might be particularly useful for developing improved medical interventions. One high-level reason for thinking this is that humans are mostly made of cells, and cellular processes happen at the nanoscale, suggesting that artificial nanoscale devices might be particularly useful.
  • General wealth / economic growth. On the face of it, if we can make high-performance materials and devices very cheaply, people will on average be very wealthy compared to today (whether this is a deviation from trend GDP growth will, of course, depend on when and how the technology is developed).
  • Social effects. Advanced nanotechnology could have important social effects, especially if it arrives abruptly. For example, strong economic growth might generally promote cooperation, while rapid technological changes might cause social upheaval.
  • Altering the global balance of power. Advanced nanotechnology might change the global balance of power. For example, it might dramatically shorten supply chains by enabling highly efficient production with unrefined local materials, leading to a shift in the balance of geopolitical power.
Footnotes
  1. In my opinion, the public work with the most authoritative and detailed description of the APM concept is Drexler’s 2013 book, Radical Abundance (especially chapter 10, and parts of chapters 1 and 2). ↩︎

  2. Drexler claims that (something similar to) complex APM would be able to manufacture products for $1/kg or less and with a throughput of 1kg every 3 hours or less per kg of machinery (for “$1/kg or less”, see, for example, Radical Abundance, 2013, p. 172; for “a throughput of 1kg every 3 hours or less per kg of machinery”, see Nanosystems, 1992, p. 1, 3rd bullet). I set a lower bar for my definition of consequential APM here because I think this lower bar is more than sufficient to imply an extremely impactful technology, while perhaps capturing a larger fraction of potential scenarios over the next few decades. ↩︎

  3. The more certain we are that the development of complex APM naturally implies the development of consequential APM soon afterwards, the less useful it is to distinguish these concepts (I think this partly explains why they are often bundled together). But I’m uncertain enough to find the distinction useful. ↩︎

  4. Someone who read a draft of this post mentioned they felt initially sceptical that it makes sense to think of ribosomes (or other nanoscale objects) as “machines”, and that they found the Wikipedia page on molecular machines helpful for reducing this scepticism. ↩︎

  5. In addition to showing that such machines are feasible, this suggests that one approach to engineering nanoscale machines doing mechanical assembly might be to use biological machines as the starting point.

    One approach that uses biological machines as the starting point is the approach I associate with synthetic biology, which involves making incremental changes to existing biological machines to achieve some goal (I’ll refer to this as the “synthetic biology approach” from now on, although I don’t know whether synthetic biologists would agree with this characterisation). This approach seems to be very challenging. My impression is that this is because the effect of small modifications is hard to predict, and because biological machines often stop working outside of their usual conditions. Still, I think some people see this as the best approach for developing something like advanced nanotechnology.

    Another approach that uses biological machines as the starting point is the “soft path” approach to APM described by Drexler (see, for example, Appendix II of Radical Abundance). This approach also starts with engineering biological molecules such as proteins (or synthetic but somewhat similar molecules), but is quite different to the synthetic biology approach, because the biological molecules are used more as a building material than as active machines in their own right, and are generally completely removed from their biological context. This soft path approach seems like a more plausible path to me than the synthetic biology approach. So far the soft path approach has had far less effort directed to it than the synthetic biology approach, as far as I’m aware. ↩︎

  6. I’m mostly relying on the analysis described in Paul Christiano’s Simple evolution analogy for this claim. From that document:

    I think the typical pattern is for human artifacts to be 2-3 OOM [order of magnitude, i.e. a factor of 10] less efficient than their biological analogs, measured by “How much energy/mass is needed to achieve a given level of performance?”

    For example, a top GPU is said to perform roughly 1-2 OOM fewer Flops per unit power than a human brain, and artificial photodetectors are said to require roughly 3-4 OOM as much power as a human eye to attain the same level of performance. Note that the analysis in that document is fairly rough, although I’d be very surprised if it turned out not to be the case that naturally evolved systems often outperform present-day ones on important dimensions. ↩︎

  7. For example, we could imagine a system occupying a 1000nm × 1000nm × 1000nm volume, and composed of nanoscale machines, that can perform one assembly operation at a time. We could fit 1,000,000,000,000 such systems into 1cm³. (Atomic force microscopes are used today to manipulate atoms one at a time, and my impression, from briefly talking to someone who uses atomic force microscopes for their research, is that they are generally around 1cm³ in size or larger (potentially much larger, depending on what you count as the atomic force microscope and what you count as its supporting infrastructure).) ↩︎

  8. Though note that these high frequency, parallel operations involving single molecules each make a tiny amount of progress towards creating a structure of appreciable size, so overall we might expect these machines to make their own mass in product within say minutes, hours, or days, rather than say once per second or 1 billion times per second. ↩︎

  9. According to Paul Christiano’s Simple evolution analogy: “Manufacturing costs differ by a further 2-4 OOM” (“OOM” stands for “order of magnitude”, i.e. a factor of 10). ↩︎

  10. I haven’t given a large amount of thought to how advanced nanotechnology might deviate from consequential APM, and I can easily see my views changing significantly in the future. Naively, though, here are some ways I imagine the technology might deviate from consequential APM: it might not be the case that most of the assembly machinery is atomically precise; maybe assembly methods or physical phenomena other than mechanical positioning are very important; maybe important components involved in the assembly of nanoscale products are not themselves nanoscale; maybe the assembly machines would not be very stiff; or maybe “control the trajectories of small molecules” wouldn’t be a natural description of the machinery’s operation.

    These alternatives to APM might often be things that are technically not consequential APM but are so similar that they can be treated as equivalent to consequential APM for most practical purposes. But perhaps many have substantial enough differences that the distinction turns out to matter for thinking about nanotechnology strategy. ↩︎

  11. To give a rough quantification of my (very unstable) beliefs here: if we have advanced nanotechnology in 2040, I’d guess that in roughly 80% of cases the advanced nanotechnology doesn’t look like complex APM. This would imply that, in some relevant sense, advanced nanotechnology covers 5x as much of the space of technological possibilities as does APM. ↩︎

  12. For example, we might sometimes be more interested in the question “how likely is it that advanced nanotechnology will be developed by 2040?” than the question “how likely is it that APM will be developed by 2040?”, because advanced nanotechnology has consequences as important as those of APM (or perhaps even more important), and answering the question for advanced nanotechnology might involve similar considerations but have a significantly different answer. ↩︎

  13. This guess of a 70% probability is basically driven by: “It’s hard for me to imagine this not being a really big deal, but I haven’t thought about this that much, and maybe my imagination just isn’t very good.” ↩︎

  14. For example, this transcript of an 80,000 Hours podcast with Carl Shulman quotes Shulman discussing the Soviets’ apparent willingness to develop bioweapons that they couldn’t protect their own people against (though presumably a weapon that could kill everyone would look less appealing than a weapon that kills some random fraction of the global population; and Soviet decision-makers might have felt that they could protect themselves through physical isolation, for example):

    It’s hard to know exactly how much work they would put into pandemic things, because… With pandemic pathogens, they’re going to destroy your own population unless you have already made a vaccine for it.

    And so the US eschewed weapons of that sort towards the end of its bioweapons program before it abandoned it entirely, on the theory that they only wanted weapons they could aim. But the Soviets did work on making smallpox more deadly, and defeat vaccines. So, there was interest in doing at least some of this ruin-the-world kind of bioweapons research. ↩︎

  15. We might also expect that the TAI we get might be different if it’s developed using a huge amount of computational resources made available by advanced nanotechnology (other than the differences we’d expect to directly follow from earlier TAI). For example, maybe the TAI we get would be more likely to follow present-day paradigms, like stochastic gradient descent, which (very speculatively) could have safety implications. ↩︎

  16. Although perhaps some new weapons other than grey goo would also involve nanoscale devices, and you might consider the components of new kinds of computers to be “novel nanoscale devices”. ↩︎

  17. Here’s a quick attempt to brainstorm considerations that seem to be feeding into my views here: "Drexler has sketched a reasonable-looking pathway and endpoint", "no-one has shown X isn't feasible even though presumably some people tried", "things are complicated and usually don't turn out how you expect", "no new physics is needed", "X seems intuitively doable given my intuitions from molecular simulations", "it's hard to be sure of anything", "trend-breaking tech rarely happens but does sometimes". ↩︎

  18. Richard Jones is a notable example of a highly credentialed person who seems to have engaged seriously with the question of the feasibility of APM. He seems to think that something resembling complex APM is feasible, although he seems sceptical about the feasibility of something with the kind of capabilities described by consequential APM. See, for example, Open Philanthropy’s A conversation with Richard Jones on September 30, 2014. ↩︎

  19. I’d speculate that there might be a decent minority of now-senior researchers who entered the field around 1995-2005, excited by the vision for nanotechnology laid out in Drexler’s works and the creation of the National Nanotechnology Initiative in the US. Perhaps these researchers nowadays have a vague sense for what Drexler’s ideas are (and probably consider APM-like visions for nanotechnology to be too far in the future to be worth thinking about). ↩︎

  20. Michael Nielsen has written a very nice, relatively quick introduction to scanning tunnelling microscopy (scanning tunnelling microscopy is a type of scanning probe microscopy). ↩︎

  21. My perception is that the hard path to APM is less promising than an alternative path called the “soft path” (see footnote 5). This perception mostly comes from what people around me seem to think, and those views in turn perhaps mostly come from Drexler’s view on this. I don’t have much of an inside view here myself; my impression is that deliberate hard path work has been occurring for many years (most notably at Zyvex) without much to show for it, but this seems like only weak evidence, partly because the level of investment has apparently been quite low. ↩︎

  22. For a more thorough overview of recent (and less recent) R&D relevant for advanced nanotechnology, see Adam Marblestone’s Bottleneck analysis: positional chemistry. The most relevant sections are “Building blocks that emerged in the meantime”, and “Explicit work on positionally directed covalent chemistry”. (Note that the focus of the report is on a technology the author calls “positional chemistry”, which is different to advanced nanotechnology, but is closely related.) ↩︎

  23. Although positional assembly with DNA suprastructures might soon be demonstrated, as a result of this grant, which seems to me to represent relevant progress. ↩︎

  24. Of course, this lack of empirical evidence also pushes against high confidence that progress will be very fast if a large research effort emerges. But I expect people will already be inclined not to be confident that progress will be very fast if a large research effort emerges. ↩︎

  25. To be clear, this number does not assume feasibility, unlike my median timeline estimate. ↩︎

  26. As a somewhat independent data point, Daniel Eth, a former colleague of mine at the Future of Humanity Institute who has spent time thinking about APM, told me that he guesses a 1% probability for roughly the event “advanced nanotechnology arrives by 2040 and isn't preceded by advanced AI”, where "advanced AI" is, roughly "AGI or transformative AI or CAIS or some future AI that is a similarly huge deal" (note that AI that dramatically speeds up technological progress doesn't necessarily qualify as advanced AI). Daniel said, “I'd expect [the estimate would] move around a bit, but probably not more than one order of magnitude.” ↩︎

  27. Here are some examples of work from these organisations:

    ↩︎
  28. One potential objection might be that points 1 or 2 just stem from general uncertainty about advanced nanotechnology due to a relative lack of attention on the relevant questions; for example, you might think that if we tried a bit harder to reduce our uncertainty, we would likely end up finding that there’s a sufficiently low probability that advanced nanotechnology arrives in the next 20 years without TAI that this area doesn’t seem worth investigating. A counterargument would be that the above is really an argument about which projects to prioritise in this area, rather than an argument for not thinking about this area at all. Similar comments apply to the uncertainty that is driving point 1. Of course, this counterargument only works if you think points 1 and 2 are reasonable positions given our current state of knowledge. ↩︎

  29. Note that an assumption underlying this argument is that there’s not much value in doing nanotechnology strategy work to understand or prepare for scenarios where TAI precedes the development of advanced nanotechnology, because TAI makes everything go crazy such that it’s hard to plan for after that point. However, if you do think such work might be valuable, the case for nanotechnology strategy research looks stronger, because there’s a broader range of future scenarios where this work is relevant. (As mentioned in the main text, it seems reasonable to me to think that work on nanotechnology strategy will be much less useful if advanced nanotechnology is preceded by TAI, but I’m not confident in this and my view feels quite unstable.) ↩︎

  30. Although note that, for example, Drexler has written several books on topics within advanced nanotechnology and the Foresight Institute runs a programme promoting the development of APM, so discussion of advanced nanotechnology would not happen against a background of complete silence on the topic. ↩︎

  31. This isn’t strictly true because some molecules might contain different isotopes; but it doesn’t seem likely that the presence of different isotopes would usually matter much, and rare isotopes could in principle be filtered out if necessary. ↩︎

  32. Provided that there are no errors in the assembly process that lead to incorrect building block placement or bonding. Products could be checked for errors and any errors could be corrected or the misformed products could be discarded; although it’s not clear to me how effective this would be in practice at eliminating errors. ↩︎

  33. In Biological and Nanomechanical Systems: Contrasts in Evolutionary Capacity (1989), Drexler discusses how and why designed systems differ from evolved ones. These LessWrong posts also make relevant arguments, in the context of TAI: Building brain-inspired AGI is infinitely easier than understanding the brain and Birds, Brains, Planes, and AI: Against Appeals to the Complexity/Mysteriousness/Efficiency of the Brain. ↩︎

  34. This 2016 Nature Methods article reports bacteria growing with a cell count doubling time of less than 10 minutes. I haven’t found numbers for the rate of total mass doubling, but I understand that cell mass remains roughly constant over successive generations, implying that total mass doubles at roughly the same rate as cell count. ↩︎

  35. Relatedly, Robert Freitas and Ralph Merkle report in Kinematic Self-Replicating Machines that ribosomes can produce their own mass in proteins in ~5-12 minutes (Ctrl+F for “If the bacterial ribosome” on the linked page). ↩︎

  36. Drexler describes something along these lines in The Stealth Threat: An Interview with K. Eric Drexler. ↩︎

  37. Though it’s not completely clear to me how useful the “advanced nanotechnology” framing is here, as opposed to, say, something like “advanced biotechnology”. ↩︎



Discuss

What Would It Cost to Build a World-Class Dredging Vessel in America?

2 мая, 2022 - 20:47
Published on May 2, 2022 5:47 PM GMT

I'm doing some research into questions surrounding the Foreign Dredge Act of 1906, and thought I'd experiment by throwing this out there.

For context, the 31 biggest dredging vessels were not built in America, and thus cannot be used in America by law. We only have a small number of less capable vessels, and they often get redirected to short-term emergency tasks. This is preventing us from doing a bunch of very valuable things, like repairing or expanding ports, which end up taking much more time and money or not happening at all.

This podcast is recommended. You can find a transcript here. They claim that there's no way America will be able to have such capacity for at least decades. I want to verify that (and also check if any other claims here don't ring true)?

As an alternative to repealing the Dredge Act (which I'm exploring and planning to write about) another alternate would be to build world-class dredging vehicles here in America, such that they could be used. Before assuming that this is impossible, and to have a straight answer, what would happen if someone with deep pockets tried to commission a world-class dredging ship that would qualify? Could be done? Are there other impossible barriers to solve? How much would it cost and how much more would that be than building it elsewhere? How long would it take? 



Discuss

So has AI conquered Bridge ?

2 мая, 2022 - 19:59
Published on May 2, 2022 3:01 PM GMT

Bridge, like most card games, is a game of incomplete information. It is a game of many facets, most of which will have to remain unstated here. However the calculation of probabilities, and the constant revision of probabilities during play, is a key feature. Much the same can be said of poker and of certain other card games. But bridge is the game I know and I am still learning from, 45 years after I first got infected with the bridge bug

We have Alpha Go and Alpha Zero but the wizards at DeepMind have not yet seen fit to release an Alpha Bridge. Poker is played to high expert standard by specialist AI’s such as Pluribus but bridge  playing computers up till today remain well short of that. Those commonly available play parts of the game extremely well, usually making mechanistic decisions correctly yet also make quite surprising and often comical errors. It has been noted bridge would seem to be to be an excellent training ground for AGI, in ways chess and go are not.

The French company NUKKAI has just created somewhat of an earthquake by revealing a bridge playing AI called Nook that, it is claimed, can beat the world’s best players. So has bridge finally been conquered? This post will show this is not yet the case. I hope also to give a flavour of the type of probabilistic problems that routinely need to be solved at the bridge table, by AI or human, where Nook may score over a human and why, and where its performance may be somewhat inflated due to some features specific to the training and test set-up. I’ve tried to write with the non-player in mind and I proffer apologies in advance if I have failed to do that adequately.

NUKKAI have published on You Tube a 5hr commentary on entire event. To my shame my poor French renders me unable to follow it closely and I’d welcome comments from French speakers who have followed it, especially as they may be able to back-up or possibly even refute some of the comments I make. Commentaries in English are also starting to become available. Kit Woolsey, a well known expert, is publishing in depth analyses of the most interesting hands on BridgeWinners.

A word about the mechanics of the game. Please feel free to skip this section if you are familiar with it. The game is a trick taking game played with a full deck of 52 cards, dealt equally between four players, who make up two partnerships sitting in opposition. There are two parts to each hand. The first part is the Auction in which the pairs vie to decide the Contract, the number of tricks greater than six that one side or the other contract to make with one suit chosen as trumps (playing with no trump suit is also an option). The side that wins the Auction are the Declaring side. The second part is the Play of the Hand. The player to the left of the Declarer (which of the pair on the Declaring side is the Declarer is decided by a slightly arcane method I won’t go into) leads a card and the partner of the Declarer lays his  hand down face upward and takes no further part in the hand. His cards are played, in correct rotation, by the Declarer. The other two players at the table ( the Defenders) have to cooperate (within the rules of the game!) to take enough tricks to defeat the Contract. All three players are  able to have sight of exactly half the cards, their own hand, and the one on the table (The Dummy). The usual rules for trick-taking games apply. Players must follow suit if they can. Aces are high. The highest card played in the suit led to the trick wins the trick, unless there is a trump suit and a player who cannot follow suit can play a trump to win the trick. The hand that wins the trick leads to the next one.

At the end of play of all 52 cards (13 tricks) the contract will either be satisfied or it is defeated. Scores are assigned and we move on to the next hand.

Firstly let us note that the only part of the game Nook has been trained to do is play Declarer. No bidding, no defence. No collaboration with a partner required. So it is, at best, very good at only one third of the game. The set up is the Human/AI play Declarer against two of the top current bridge playing computers. It is an important point that there is always this constancy of silicon opposition, who might be expected to (and apparently do) always defend in a consistent and predictable manner. 8 experts each played 100 hands so the full test was over 800 hands. However the auction and the contract was always the same (the contract was 3NT, 9 tricks to be made with no trump suit) and there was essentially no information available in the auction to influence declarer play. So it would be very wrong to say the test evaluated AI skill over the full gamut of declarer play.

After the initial lead, it is normal for a Declarer to make a strategic plan. A good plan will optimise the likelihood of enough tricks being made to achieve the contract. Sometimes a good plan will be ultra cautious, guarding against unlikely distributions to ensure the contract is made. On other occasions, if the contract is ambitious, a risky plan may be required to have any chance of achieving the contract.

This part of the operation would seem to be relatively straightforward for a good AI to master. Finding a plan that works best against the largest number of distributions does not seem too dissimilar to the planning required in chess and go. Commentators have indeed identified cases where Nook’s apparent plan was superior to that of the human opponent.

After the initial lead by opponent/ partner 25 cards remain unseen and their positions between the two other players often need to be inferred to achieve optimum play. There is often information available from the Auction that provides an immediate Baysian update (although not in the test of Nook). Thereafter the play of any unseen card may allow another Baysian update. If we don’t count the  triviality of the last trick, this provides  possibly 23 instances where a Baysian update may be applied (and this doesn’t include any inferences to be drawn from the cards declarer chooses to play from dummy as well).

Each Baysian update may require a change in the plan. This process of making inferences from opponents actions and gathering additional information, to adjust prior probabilities assigned to several different card distributions ( we could also call them models) is called “Card Reading”, a term with somewhat mystical association. When carried out successfully, and the actual distribution is found to match the predicted one, it does feel a bit like magic.

The most informative case is where a defender can’t follow suit. Immediately the distribution of that suit is known and the likely distributions of the other suits becomes altered. But more subtle inferences can also be derived from the nature of cards played to follow suit, or chosen as discards.  

It is unlikely that all possible eventualities are taken account of in the original plan, the combinatorial explosion of possible distributions is too vast. So Nook most likely reassesses the plan after each card, similar to a human player.


At this stage let’s look at some simple situations where probabilistic reasoning can be applied and how an AI might need to reason. Here is quite a common type of position in a single suit.
 

Dummy:   AJ2

Declarer:  K103

Declarer wants to make 3 tricks. She could first play the K,and then lead the 3 to the J, winning if the Q is on the left (should the left hand player play the Q it will be topped by the A). Or she could lead the 3 to the A and, on the next trick, lead the 2 towards the 10. This wins if the Q is on the right. A priori it is a guess, a 50:50 proposition which play to choose. However an observant player will generally have additional information that adjusts the probabilities one way or the other. This information can be gained from the Auction, or during play. For instance, declarer may know from the Auction that the left hand player is highly likely to have  6 cards in another suit and, knowing the declaring partnership has 5 cards in that same suit, right hand player therefore has 2 of that suit. Left Hand, starting with thirteen cards, therefore has at the outset 7 “empty spaces“ and Right Hand 11 “empty spaces” in which to put the Q. All other things being equal, the chance of the Q of this suit being on the Right is now favoured to the extent of 11/18.

      A good player will not be content with that. She will play other suits to test their distribution and the disposition of high cards, with a view to learning more about the hand. Sometimes this will not be productive but sometimes it will harden the odds, making it more likely the Q is on the Right. More rarely the picture will change completely and odds start to favour the Q being on the Left. Sometimes certainty can be achieved and the position of the key card can be identified with probability 1.

It would seem clear that an AI should be able to recalculate these probabilities faster and more accurately than a human can. Superiority in this part of the game would not be surprising. Whether Nook can plan the play to elicit as much information from the opposition as possible is an interesting question.

As far as I can ascertain the computer defenders do play in a deterministic fashion. Their manner of play of the low cards in any given situation will be consistent.  There does appear to be some indication that Nook may indeed seek out additional information to a certain extent, relying on the deterministic play of the defenders to build up a true picture  (human defenders would rarely be as accommodating). More on this later.

Here is an example that is sometimes quoted as classically Baysian. It is somewhat infamous, in that the principle, the so called “Principle of Restricted Choice”, is, in my experience, not accepted (sometimes vehemently so) by some bridge players. Although unintuitive at first glance, the logic behind it is correct, however.

Dummy: A10432

Declarer:  K765

There are 4 cards missing In the suit. Suppose, to the first trick, you play the K and see Left Hand play the 8  (you play small from Dummy) and Right Hand plays the Q. What is the play that gives the best chance of not losing a trick in the suit?

    There are two holdings that are consistent with Right Hand’s play of the Q. Q on its own (singleton) or QJ doubleton.  If the first holding is correct you need to play a low card to the 10, finessing Left Hand’s J. If the second is correct you need to play to the A. The two holdings are roughly equal in likelihood, given no further information. So, it must be the same as a toss of a coin surely?

    No, it isn’t. With no other information available, the play to the 10 figures to win two thirds of the time. The reason is that, with QJ, Right Hand has a choice of two equivalent plays on the first trick, there being no material effect on the outcome of  playing the Q rather than the J. Consequently the chance of the Q being originally in the holding QJ, is one half that of the Q being singleton, where there is no choice of play.

    This situation is exactly equivalent to the famous Monty Hall problem. The case where the prize is behind your door corresponds to that of when Right Hand has QJ. The Host can choose which of the other two doors to open. The case where the prize is behind one of the other two doors corresponds to the singleton Q case. The host has no choice which door to open, without revealing the prize. You should switch doors.

The principle of restricted choice is most well known for examples involving just high cards, similar to that above. However in principle it can often be applied in some form or other in many situations. Asking the question of whether a play is forced, or is one of several equal choices, as often as possible, even regarding the smaller cards, should in theory allow more accurate assessment of which are the likely distributions. This requires great attention to detail, accurate sifting of signal from noise, and a strong and quick mental calculation ability, and  few players are capable of sustaining the effort to do this consistently for a great length of time. Having said that, very good players, through long experience, do, almost unconsciously, pick up on clues that would be insignificant to weaker players and draw inferences on them.

There is a wrinkle that is important to mention in regard to this example. Suppose we have some extra information, namely that the agent on the right will always play the Q from QJ. There is no possible benefit to do so in this situation. However there is a common convention that, with two cards only, a defender will play the highest, so long as it doesn’t cost, thereby giving information to their partner as to their length in the suit. Players can be creatures of habit, and apply that convention when there is no need. This changes the odds. Now the play of the 10 is back to being close to a 50-50 choice. However, on the flip side, had Right Hand played the J rather than the Q, it would now be a near certainty that no other card accompanies it. Note that our consistent robot defenders are creatures of habit and are likely always play the same card from QJ doubleton. They are also likely to (say) always play low from three small, play high from doubleton etc etc. and thus signal the distribution of their cards accurately.

This introduces the point that each opponent has different experience, different strengths and weaknesses, different quirks and habits. The best players (including AI’s) need to take these into account if they can to accurately infer the probabilities of various holdings, from their play. 

Nook may be able to gain significant advantage in these type of situations by constantly  eking out  Baysian inferences from the small card plays that even the best human players don’t consistently do. Also, crucially, Nook will eke out significant information in from  the consistent play of defenders. Human defenders, by contrast, do not in general play consistently, thereby generating noise, and human declarers do not usually generally learn the playing style of individual opponents in great detail, as in most competitions, they will play against many opponents.

Comparative analysis of differential human/Nook strategies is still on going but evidence is emerging that Nook’s mastery may to a considerable extent be due to its exploitation of the behaviour of the defenders. Not only do they signal their distribution in a rigorous fashion but they make errors of play in a consistent fashion, which may very well be exploitable. It looks like Nook has learnt very well how to exploit the foibles of its opposition.

Let’s look at another card layout.

Dummy: Q103

Declarer  : A42

Declarer wants two tricks in the suit. A normal play here is to lead low from declarers’s hand and play either the Q or the 10 when a low card is played on the Left. If the K and J are on the Left either will do. If the K and J are on the Right neither will work. If the K is on the L and J on the R it is necessary to put in the Q. If the J is on the L and K on the R  it is necessary to to put in the 10. A 50% proposition? No, not completely. A good player will usually put in the 10. The consideration is what Left Hand Player didn’t do on this trick, which is to put in the K. With K and only one other card, most average players would play the K to make sure they make a trick with it instead of it being lost under the A. So this holding is now unlikely. With K and 2 or more cards it is more normal to play low but some players might still find a reason to play the K. With J and one other card ( or more ) it would always be normal to play low in front of dummy. The consideration  that the holding of Kx  on the Left side ( where x denotes a small card) is unlikely, is enough to tweak the probabilities and make the 10 the right play. (Note: Before it is pointed out to me, yes, playing the A first before playing to the 10 would reveal immediately if Left Hand had Kx. In some cases this would be the right play. Here, let us assume declarer has other options for an extra trick and cannot afford to put all his eggs in one basket by risking the loss of two immediate tricks)

This is firstly, an  example of drawing an inference from what hasn’t happened ( c.f. Sherlock Holmes’ dog that didn’t bark in the night). This is, in my view, one of the more difficult rationality techniques. The best bridge players are adept at this. The rest of us find this hard to do.

It will be interesting to find out how Nook handles this type of situation, Does such inferential thinking get encoded naturally during the training regime?

Secondly, it is an example where the Declarer has to take into account  how another agent at the table is likely to play, given a certain holding. It is becoming clear that Nook has become adapt at this, albeit only because it has trained against a single deterministic opponent.

A last point, this is also an example of how the probabilities can change depending on the agents at the table. This situation offers a very good player (or exceptional AI) in the Left Hand seat an opportunity. With Kx she might very well quickly play low, (taking time to make that decision is not an option, it signals where the K is). She knows that Declarer doesn’t have AJx as otherwise he would likely be playing the Q or 10 towards the AJx to try to make three tricks if the K is on the other side. So playing low is not in fact likely to cost and will very likely induce a missed guess, the 10 losing to the J. The pay-back occurs if declarer is dealt A9x, as then later Declarer is likely to play from Dummy towards the 9 and lose a second trick to the lone K.

We have discussed how a human or AI can maximise their own performance. However good players must also try to minimise the performance of the opposition, by restricting the information they themselves give out.

For example there is a bridge aphorism ‘play the card you are known to hold”.
 

  Dummy:    432

        1098                    KJ7

 Declarer:    AQ65
 

Declarer plays the 4 from dummy and, R Hand playing the 7, puts in the Q, winning the trick. Now he cashes the A and Right Hand drops the K, the card he is known to have. Now declarer has no more  information. He can play a third round of the suit to establish a winner, but he might suspect the suit is breaking badly. It is easier for declarer to place the cards if E plays the J rather than the K.

This we can call obfuscation.  Taken to extreme, obfuscation becomes deception, with the distinction, should we want one, that an obfuscation is generally zero cost play, whereas a deceptive play should have some jeopardy if the deception is challenged.

Here is an example of deception by a defender.

Dummy:   AQ109

Declarer:   32

It is normal here for Declarer to play low to the 10 or 9 , and, if it loses to the J, later play towards the Q or 9. There are two bites of the cherry at getting three tricks. On a good  day, four tricks are possible. On this occasion Right Hand wins first time with the K. Later Declarer plays confidently towards the 9 and is dismayed to see it lose to the J that was ‘marked’ as being in the opposite hand. Perhaps now the AQ are stranded without access and cannot  make tricks. Or perhaps declarer has in the meantime burned his boats elsewhere and staked all on the expectation of three tricks in this suit.

This is a lovely coup to bring off. But it is a gamble. If declarer sees through the stratagem you are likely to be considerably worse off.

Opportunities for true deception are more commonly found in defence play but can also be found in declarer play. Poker playing robots have been demonstrated to bluff effectively so in theory an exceptional bridge playing AI might also be able to play deceptively. However  the AI has to recognise the relatively rare situations where deception is the optimum tactic and it is not clear how easy that is to train.

If there are poker and bridge playing robots that can deceive their human opponents, the question arises as whether they are actually unaligned, and, if so, whether they are unaligned in a trivial sense or in a fundamental sense that may be be important and useful to AI alignment researchers. This may be a well trod topic, for all I know, but if not, it is worthy of discussion.

It is time to summarise:

A) Nook has beaten eight out of eight human experts in one relatively specialised aspect of bridge play. Despite the small domain space and some additional caveats this does look an impressive achievement.  Nook’s play was by no stretch of the imagination error free but it did on multiple occasions clearly locate the superior line of play.

B) Nook needs to plan a general line of play at the beginning that is flexible enough to take account of as many opposing distributions as possible. It is not surprising, given the state of the art, that Nook should be at least as good as a human at this.

C) Nook needs to make Bayesian inferences at each play of a card, reassess the plan and possibly change the plan if necessary. It seems likely Nook may have the advantage at going deeper than a human would in this regard, through always taking note of every card played and making Baysian inferences not just from the play of the high cards but also of the low cards.

D) Consideration of the expertise and style of play of the defenders may change a chosen line of play. Here, it seems that Nook has been trained against the same bridge robots that it is tested against and that these robots play in a deterministic fashion. If so Nook will have learnt the robots style of card play extremely well, and the typical errors that these computers  make. It will be able to therefore read the defenders cards very well and may well have learnt to play in a way to promote the errors peculiar to these opponents. This might give Nook a very significant yet rather artificial advantage over a human who is trained against many different opponents. This advantage would seem to be enough to explain its success rate. It is this possibility that casts the most significant question mark on the world beating claims made by NUKKAI. More analysis is needed here and it will be interesting to hear how NUKKAI themselves reply to this point.

E) Ideally Nook should play in a way that disguises it’s own cards as much as possible from the opponents. It will be interesting to find out if it does play in this way. Can it also exploit situations where it is possible to deceive the defenders, albeit at risk, and gain from it? That would an exceptionally impressive achievement in my view.

F) Bridge has not been conquered by AI ….. yet!

A final note to you non-players. Bridge is an entrancing and life enhancing game but is rapidly becoming an old persons game, because the young now have many other gaming distractions to tempt them and new young blood is now a trickle where it used to be a flood. Yet for the aspiring rationalist I can think of no better training regime for Baysian thinking than the game of bridge. The examples I have presented represent a very small glimpse of the type of thinking required at the bridge table. If they have piqueued your interest, perhaps think about finding some like-minded friends and starting a game, or looking up a local bridge teacher and signing on for a course. 



Discuss

Squires

2 мая, 2022 - 06:36
Published on May 2, 2022 3:36 AM GMT

Last year, johnswentworth posted The Apprentice Experiment. I tried it out. It was a disaster for me and most of my "apprentices". Since then I have been figuring out how to make it work. I currently have two minions. Thanks to them, I finally feel like a proper supervillain.

"How can I live without a human being to pour me drinks and fetch my dirty sandals?" I said.

"You got used to having a servant scarily fast," [Redacted] said.

―[April was weird]

For starters, I don't use the word "apprentice". The word "apprentice" is pretentious and inaccurate. A teenager's parents pay the master to accept their son[1] as an apprentice. Apprenticeship is like sending a kid to a vocational school. Would I accept apprentices? Possibly. But if you (or your parents) are not paying me for the privilege of my tutelage then you are not an apprentice.

Squires are different. Squires are not students. I don't try to teach my squires anything at all. I just give them the boring tedious work I don't want to do. Surprisingly (to me) they love it.

Teenagers want to be useful. But teenagers have limited skills. I put my squires to work doing the most complicated tasks they are capable of. Today, that meant sweeping the floor. Why is sweeping the floor fun? Because doing the most complicated work you are capable of is fun. Less complicated work is boring. More complicated work is frustrating.

"Fun" is evolution rewarding you for learning optimally.

My squires learn quickly. Last month a squire offered to spellcheck my blog posts. This month, his assignment is to negotiate a business deal with <wearables company> instead. Will he succeed? I don't know. I have better things to do than micromanage my minions. This brings me to the most important trait a squire can have.

Good squires invent work for themselves to do. If the work is valuable (like spellchecking my posts) then I tell the squire to continue doing it. If the work is worthless then I give the squire a useful assignment to do instead. "I want to be your squire. What should I do?" is a bad way to introduce yourself. "I want to be your squire. I can do x.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} " for you is a good way to introduce yourself even if x turns out to be worthless.

The most important trait a squire can have is "eager to work". A squire who is eager to work can usually made useful.

Squiring is for energetic young people who have few skills and no money. I you have money then just hire me as a personal coach instead.

  1. ^

    Middle Ages Europe was a sexist place.



Discuss

[Linkpost] A conceptual framework for consciousness

2 мая, 2022 - 04:05
Published on May 2, 2022 1:05 AM GMT

PNAS Paper from April 29th that makes strides to solve the hard problem of consciousness by dissolving it:

A conceptual framework for consciousness

by Michael S. A. Graziano

Abstract:

This article argues that consciousness has a logically sound, explanatory framework, different from typical accounts that suffer from hidden mysticism. The article has three main parts. The first describes background principles concerning information processing in the brain, from which one can deduce a general, rational framework for explaining consciousness. The second part describes a specific theory that embodies those background principles, the Attention Schema Theory. In the past several years, a growing body of experimental evidence—behavioral evidence, brain imaging evidence, and computational modeling—has addressed aspects of the theory. The final part discusses the evolution of consciousness. By emphasizing the specific role of consciousness in cognition and behavior, the present approach leads to a proposed account of how consciousness may have evolved over millions of years, from fish to humans. The goal of this article is to present a comprehensive, overarching framework in which we can understand scientifically what consciousness is and what key adaptive roles it plays in brain function.

 

Key quotes:

Principle 1.

Information that comes out of a brain must have been in that brain. [...]

For example, if I believe, think, and claim that an apple is in front of me, then it is necessarily true that my brain contains information about that apple. Note, however, that an actual, physical apple is not necessary for me to think one is present. If no apple is present, I can still believe and insist that one is, although in that case I am evidently delusional or hallucinatory. In contrast, the information in my brain is necessary. Without that information, the belief, thought, and claim are impossible, no matter how many apples are actually present.

[...]

You believe you have consciousness because of information in your brain that depicts you as having it. [...] The existence of an actual feeling of consciousness inside you, associated with the color, is not necessary to explain your belief, certainty, and insistence that you have it. Instead, your belief and claim derive from information about conscious experience. If your brain did not have the information, then the belief and claim would be impossible, and you would not know what experience is, no matter how much conscious experience might or might not “really” be inside you.

[...]

Note that principle 1 does not deny the existence of conscious experience. It says that you believe, think, claim, insist, jump up and down, and swear that you have a conscious feeling inside you, because of specific information in your brain that builds up a picture of what the conscious feeling is. [...]

Principle 2.

The brain’s models are never accurate.

...and I think you can anticipate what follows. I recommend reading the article though.



Discuss

What DALL-E 2 can and cannot do

2 мая, 2022 - 02:51
Published on May 1, 2022 11:51 PM GMT

I got access to DALL-E 2 earlier this week, and have spent the last few days (probably adding up to dozens of hours) playing with it, with the goal of mapping out its performance in various areas – and, of course, ending up with some epic art. 

Below, I've compiled a list of observations made about DALL-E, along with examples. If you want to request art of a particular scene, or to test see what a particular prompt does, feel free to comment with your requests. 

DALL-E's strengths Stock photography content 

It's stunning at creating photorealistic content for anything that (this is my guess, at least) has a broad repertoire of online stock images – which is perhaps less interesting because if I wanted a stock photo of (rolls dice) a polar bear, Google Images already has me covered. DALL-E performs somewhat better at discrete objects and close-up photographs than at larger scenes, but it can do photographs of city skylines, or National Geographic-style nature scenes, tolerably well (just don't look too closely at the textures or detailing.) Some highlights: 

  • Clothing design: DALL-E has a reasonable if not perfect understanding of clothing styles, and especially for women's clothes and with the stylistic guidance of "displayed on a store mannequin" or "modeling photoshoot" etc, it can produce some gorgeous and creative outfits. It does especially plausible-looking wedding dresses – maybe because wedding dresses are especially consistent in aesthetic, and online photos of them are likely to be high quality? 
a "toga style wedding dress, displayed on a store mannequin"
  • Close-ups of cute animals. DALL-E can pull off scenes with several elements, and often produce something that I would buy was a real photo if I scrolled past it on Tumblr.
"kittens playing with yarn in a sunbeam"
  • Close-ups of food. These can be a little more uncanny valley – and I don't know what's up with the apparent boiled eggs in there – but DALL-E absolutely has the plating style for high-end restaurants down.
"dessert special, award-winning chef five star restaurant, close-up photograph"
  • Jewelry. DALL-E doesn't always follow the instructions of the prompt exactly (it seems to be randomizing whether the big pendant is amber or amethyst) but the details are generally convincing and the results are almost always really pretty. 
"silver statement necklace with amethysts and an amber pendant, close-up photograph" Pop culture and media 

DALL-E "recognizes" a wide range of pop culture references, particularly for visual media (it's very solid on Disney princesses) or for literary works with film adaptations like Tolkien's LOTR. For almost all media that it recognizes at all, it can convert it in almost-arbitrary art styles. 

"art nouveau stained glass window depicting Marvel's Captain America""Elsa from Frozen, cross-stitched sampler"Sesame Street, screenshots from the miyazaki anime movie

[Tip: I find I get more reliably high-quality images from the prompt "X, screenshots from the Miyazaki anime movie" than just "in the style of anime",  I suspect because Miyazaki has a consistent style, whereas anime more broadly is probably pulling in a lot of poorer-quality anime art.]

Art style transfer

Some of most impressively high-quality output involves specific artistic styles. DALL-E can do charcoal or pencil sketches, paintings in the style of various famous artists, and some weirder stuff like "medieval illuminated manuscripts". 

"a monk riding a snail, medieval illuminated manuscript"

IMO it performs especially well with art styles like "impressionist watercolor painting" or "pencil sketch", that are a little more forgiving around imperfections in the details.  

"A woman at a coffeeshop working on her laptop and wearing headphones, painting by Alphonse Mucha""a little girl and a puppy playing in a pile of autumn leaves, photorealistic charcoal sketch" Creative digital art

DALL-E can (with the right prompts and some cherrypicking) pull off some absolutely gorgeous fantasy-esque art pieces. Some examples: 

"a mermaid swimming underwater, photorealistic digital art""a woman knitting the Milky Way galaxy into a scarf, photorealistic digital art"

The output when putting in more abstract prompts (I've run a lot of "[song lyric or poetry line], digital art" requests) is hit-or-miss, but with patience and some trial and error, it can pull out some absolutely stunning – or deeply hilarious – artistic depictions of poetry or abstract concepts. I kind of like using it in this way because of the sheer variety; I never know where it's going to go with a prompt. 

"an activist destroyed by facts and logic, digital art""if the lord won't send us water, well we'll get it from the devil, digital art""For you are made of nebulas and novas and night sky You're made of memories you bury or live by, digital art" (lyric from Never Look Away by Vienna Teng)The future of commercials 

This might be just a me thing, but I love almost everything DALL-E does with the prompt "in the style of surrealism" – in particular, its surreal attempt at commercials or advertisements. If my online ads were 100% replaced by DALL-E art, I would probably click on at least 50% more of them. 

"an advertisement for sound-cancelling headphones, in the style of surrealism"DALLE's weaknesses

I had been really excited about using DALL-E to make fan art of fiction that I or other people have written, and so I was somewhat disappointed at how much it struggles to do complex scenes according to spec. In particular, it still has a long way to go with:

Scenes with two characters 

I'm not kidding. DALL-E does fine at giving one character a list of specific traits (though if you want pink hair, watch out, DALL-E might start spamming the entire image with pink objects). It can sometimes handle multiple generic people in a crowd scene, though it quickly forgets how faces work. However, it finds it very challenging to keep track of which traits ought to belong to a specific Character A versus a different specific Character B, beyond a very basic minimum like "a man and a woman." 

The above is one iteration of a scene I was very motivated to figure out how to depict, as a fan art of my Valdemar rationalfic. DALL-E can handle two people, check, and a room with a window and at least one of a bed or chair, but it's lost when it comes to remembering which combination of age/gender/hair color is in what location. 

"a young dark-haired boy resting in bed, and a grey-haired older woman sitting in a chair beside the bed underneath a window with sun streaming through, Pixar style digital art"

Even in cases where the two characters are pop culture references that I've already been able to confirm the model "knows" separately – for example, Captain America and Iron Man – it can't seem to help blending them together. It's as though the model has "two characters" and then separately "a list of traits" (user-specified or just implicit in the training data), and reassigns the traits mostly at random.

"Captain America and Iron Man standing side by side" which is which????Foreground and background

A good example of this: someone on Twitter had commented that they couldn't get DALL-E to provide them with "Two dogs dressed like roman soldiers on a pirate ship looking at New York City through a spyglass". I took this as a CHALLENGE and spent half an hour trying; I, too, could not get DALL-E to output this, and end up needing to choose between "NYC and a pirate ship" or "dogs in Roman soldier uniforms with spyglasses". 

DALL-E can do scenes with generic backgrounds (a city, bookshelves in a library, a landscape) but even then, if that's not the main focus of the image then the fine details tend to get pretty scrambled. 

Novel objects, or nonstandard usages 

Objects that are not something it already "recognizes." DALL-E knows what a chair is. It can give you something that is recognizably a chair in several dozen different art mediums. It could not with any amount of coaxing produce an "Otto bicycle", which my friend specifically wanted for her book cover. Its failed attempts were both hilarious and concerning. 

prompt was something like "a little girl with dark curly hair riding down a barren hill on a magical rickshaw with enormous bicycle wheels, in the style of Bill Watterson"An actual Otto bicycle, per Google Images

Objects used in nonstandard ways. It seems to slide back toward some kind of ~prior; when I asked it for a dress made of Kermit plushies displayed on a store mannequin, it repeatedly gave me a Kermit plushie wearing a dress. 

"Dress made out of Kermit plushies, displayed on a store mannequin"

DALL-E generally seems to have extremely strong priors in a few areas, which end up being almost impossible to shift. I spent at least half an hour trying to convince it to give me digital art of a woman whose eyes were full of stars (no, not the rest of her, not the background scenery either, just her eyes...) and the closest DALL-E ever got was this.

I wanted: the Star-Eyed Goddess
I got: the goddess-eyed goddess of recursionSpelling

DALL-E can't spell. It really really cannot spell. It will occasionally spell a word correctly by utter coincidence. (Okay, fine, it can consistently spell "STOP" as long as it's written on a stop sign.) 

It does mostly produce recognizable English letters (and recognizable attempts at Chinese calligraphy in other instances), and letter order that is closer to English spelling than to a random draw from a bag of Scrabble letters, so I would guess that even given the new model structure that makes DALL-E 2 worse than the first DALL-E, just scaling it up some would eventually let it crack spelling.  

At least sometimes its inability to spell results in unintentionally hilarious memes? 

EmeRAGEencey!Realistic human faces

My understanding is that the face model limitation may have been deliberate to avoid deepfakes of celebrities, etc. Interestingly, DALL-E can nonetheless at least sometimes do perfectly reasonable faces, either as photographs or in various art styles, if they're the central element of a scene. (And it keeps giving me photorealistic faces as a component of images where I wasn't even asking for that, meaning that per the terms and conditions I can't share those images publicly.) 

Even more interestingly, it seems to specifically alter the appearance of actors even when it clearly "knows" a particular movie or TV show. I asked it for "screenshots from the second season of Firefly", and they were very recognizably screenshots from Firefly in terms of lighting, ambiance, scenery etc, with an actor who looked almost like Nathan Fillion – as though cast in a remake that was trying to get it fairly similar – and who looked consistently the same across all 10 images, but was definitely a different person. 

There are a couple of specific cases where DALL-E seems to "remember" how human hands work. The ones I've found so far mostly involve a character doing some standard activity using their hands, like "playing a musical instrument." Below, I was trying to depict a character from A Song For Two Voices who's a Bard; this round came out shockingly good in a number of ways, but the hands particularly surprised me. 

Limitations of the "edit" functionality 

DALL-E 2 offers an edit functionality – if you mostly like an image except for one detail, you can highlight an area of it with a cursor, and change the full description as applicable in order to tell it how to modify the selected region. 

It sometimes works - this gorgeous dress (didn't save the prompt, sorry) originally had no top, and the edit function successfully added one without changing the rest too much.

This is how people will dress in the glorious transhumanist future. 

It often appears to do nothing. It occasionally full-on panics and does....whatever this is. 

I was just trying to give the figure short hair!

There's also a "variations" functionality that lets you select the best image given by a prompt and generate near neighbors of it, but my experience so far is that the variations are almost invariably less of a good fit for the original prompt, and very rarely better on specific details (like faces) that I might want to fix.

Some art style observations 

DALL-E doesn't seem to hold a sharp delineation between style and content; in other words, adding stylistic prompts actively changes the some of what I would consider to be content. 

For example, asking for a coffeeshop scene as painted by Alphonse Mucha puts the woman in in a long flowing period-style dress, like in this reference painting, and gives us a "coffeeshop" that looks a lot to me like a lady's parlor; in comparison, the Miyazaki anime version mostly has the character in a casual sweatshirt. This makes sense given the way the model was trained; background details are going to be systematically different between Nouveau Art paintings and anime movies. 

"A woman at a coffeeshop working on her laptop and wearing headphones, painting by Alphonse Mucha""A woman at a coffeeshop working on her laptop and wearing headphones, screenshots from the miyazaki anime movie"

DALL-E is often sensitive to exact wording, and in particular it's fascinating how "in the style of x" often gets very different results from "screenshot from an x movie". I'm guessing that in the Pixar case, generic "Pixar style" might capture training data from Pixar shorts or illustrations that aren't in their standard recognizable movie style. (Also, sometimes if asked for "anime" it gives me content that either looks like 3D rendered video game cutscenes, or occasionally what I assume is meant to be people at an anime con in cosplay.) 

"A woman at a coffeeshop working on her laptop and wearing headphones, screenshots from the Pixar movie""A woman at a coffeeshop working on her laptop and wearing headphones, in the style of Pixar"Conclusions

How smart is DALL-E? 

I would give it an excellent grade in recognizing objects, and most of the time it has a pretty good sense of their purpose and expected context. If I give it just the prompt "a box, a chair, a computer, a ceiling fan, a lamp, a rug, a window, a desk" with no other specification, it consistently includes at least 7 of the 8 requested objects, and places them in reasonable relation to each other – and in a room with walls and a floor, which I did not explicitly ask for. This "understanding" of objects is a lot of what makes DALL-E so easy to work with, and in some sense seems more impressive than a perfect art style. 

The biggest thing I've noticed that looks like a ~conceptual limitation in the model is its inability to consistently track two different characters, unless they differ on exactly one trait (male and female, adult and child, red hair and blue hair, etc) – in which case the model could be getting this right if all it's doing is randomizing the traits in its bucket between the characters. It seems to have a similar issue with two non-person objects of the same type, like chairs, though I've explored this less. 

It often applies color and texture styling to parts of the image other than the ones specified in the prompt; if you ask for a girl with pink hair, it's likely to make the walls or her clothes pink, and it's given me several Rapunzels wearing a gown apparently made of hair. (Not to mention the time it was confused about whether, in "Goldilocks and the three bears", Goldilocks was also supposed to be a bear.) 

The deficits with the "edit" mode and "variations" mode also seem to me like they reflect the model failing to neatly track a set of objects-with-assigned-traits. It reliably holds the non-highlighted areas of the image constant and only modifies the selected part, but the modifications often seem like they're pulling in context from the entire prompt – for example, when I took one of my room-with-objects images and tried to select the computer and change it to "a computer levitating in midair", DALL-E gave me a levitating fan and a levitating box instead. 

Working with DALL-E definitely still feels like attempting to communicate with some kind of alien entity that doesn't quite reason in the same ontology as humans, even if it theoretically understands the English language. There are concepts it appears to "understand" in natural language without difficulty – including prompts like "advertising poster for the new Marvel's Avengers movie, as a Miyazaki anime, in the style of an Instagram inspirational moodboard", which would take so long to explain to aliens, or even just to a human from 1900. And yet, you try to explain what an Otto bicycle is – something which I'm pretty sure a human six-year-old could draw if given a verbal description – and the conceptual gulf is impossible to cross. 

"advertising poster for the new Marvel's Avengers movie, as a Miyazaki anime, in the style of an Instagram inspirational moodboard"

Discuss

ELK shaving

2 мая, 2022 - 01:51
Published on May 1, 2022 9:05 PM GMT

> Paul Christiano's incredibly complicated schemes have no chance of working in real life before DeepMind destroys the world. > Eliezer in Death With Dignity

 

Eliciting Latent Knowledge reads to me as an incredibly narrow slice in reasoning space, a hyperbolically branching philosophical rabbit hole of caveats.

 

For example, this paragraph on page 7translates as:

If you can ask whether AI is saying the truth and it answers "no" then you know it is lying.

But how is trusting this answer different from just trusting AI to not deceive you as a whole?

A hundred pages of an elaborate system with competing actors playing games of causal diagrams trying to solve for the worst case is exciting ✨ precisely because it allows one to "make progress" and have incredibly nuanced discussions (ELK shaving [1]) while failing to address the core AI safety concern:

if AI is sufficiently smart, it can do absolutely whatever

– fundamentally ignoring whatever clever constraints one might come up with.

I am confused why people might be "very optimistic" about ELK, I hope I am wrong.



 

  1. ^“Yak shaving” means performing a seemingly endless series of small tasks that must be completed before the next step in the project can move forward. Elks are kinda like yaks


Discuss

Is it desirable for the first AGI to be conscious?

2 мая, 2022 - 00:29
Published on May 1, 2022 9:29 PM GMT

It seems that consciousness is one of the essential things necessary to characterize the moral status of an agent.
It seems that we have very little chance to solve the AI safety problem.
We don't know if the first AGI will be conscious.

Since we are in a certain sense doomed, the future consists of the following 2 cases:

  • Either the first AGI is not sentient and never becomes sentient.
  • Or the first AGI acquires sentience at a given moment.
    • Either the valence of this AGI is positive
    • Or the valence of this AGI is negative
    • Either the valence is mixed
    • Or the AGI reprograms itself to experience a positive valence (instrumental convergence if its well-being is included in the utility function?)

I have no idea which scenario is the most desirable. The uncertainty is multiplied by considering the AGI's superior capabilities, and its ability to multiply. So perhaps we could see it as a utility monster. Therefore, the following questions seem very important:

(1) What do you think is the most likely scenario and (2) the most desirable scenario?

(3|2) Conditional on the scenario that seems most desirable to you, is there a way to steer the future in that direction?



Discuss

Страницы