# Новости LessWrong.com

A community blog devoted to refining the art of rationality
Обновлено: 49 минут 49 секунд назад

### [Linkpost] Hormone-disrupting plastics and reproductive health

19 октября, 2021 - 14:01
Published on October 19, 2021 11:01 AM GMT

Interview with the author

Discuss

### Are people here still looking for crypto tips?

19 октября, 2021 - 12:28
Published on October 19, 2021 9:28 AM GMT

Just saw this.

https://www.lesswrong.com/posts/MajyZJrsf8fAywWgY/a-lesswrong-crypto-autopsy

I have fairly high confidence (> 50%) of atleast doing atleast 5x returns on 1 of 2 or 3 opportunities, wondering if people are interested.

Will require you to suspend your EMH beliefs - they don't apply to illiquid tiny markets that no hedge fund is looking at. Nor do they apply to markets that have no notion of efficient or correct price based on cashflow.

Discuss

### [Book Review] "Suffering-focused Ethics" by Magnus Vinding

19 октября, 2021 - 02:34
Published on October 18, 2021 11:34 PM GMT

Magnus Vinding’s “Suffering-Focused Ethics” is a heartfelt, yet occasionally overreaching work laying out a case for why we should focus primarily on suffering in our ethical decisions, and exactly how we can approach the world with this framework.

“Suffering-Focused Ethics” is split into two sections. The first section lays the terminological and ethical framework for the second half, introducing, refining, and defending concepts such as The Asymmetry, Principle of Sympathy for Intense Suffering, Extreme Suffering, and Moral Realism. The second section utilizes these terms in more applied cause areas, such as Space Colonization, Wild Animal Suffering, Moral Cooperation, and Personal Self-care.

Given that I already agree with most of Vinding’s views, I found the first section to be relatively dry (although with a couple of quirky arguments, to be discussed), and I was much more impressed by the second section, particularly the seldom discussed case against Space Colonization.

In my review of the first section, I will first lay out Vinding’s views as I have understood them, and then more thoroughly dissect both his chapter on the Principle of Sympathy as well as his off-key chapter on Moral Realism. I will be spending a decent chunk of time on the Moral Realism chapter specifically, because I found the arguments contained within it to be particularly unique.

In my review of the second section, I will discuss and expand almost exclusively on the chapter focusing on Space Colonization and S-Risk. I will have a few notes in passing on the remaining topics.

Section 1 - The Case for Suffering-Focused EthicsChapters in Section 1

“The problem of suffering is the greatest problem of all. This is my conviction, and why I have set out to write this book, in which I will explore the why and how of reducing suffering.” (pg.1)

Vinding’s main theses in this first section may be split into roughly four:

1. Suffering and Happiness are asymmetrical (in a variety of ways), and not easily placed on the same scale,
2. Happiness consists of the reduction of suffering,
3. Positive goods cannot outweigh the worst forms of suffering, and
4. that we truly and objectively ought to reduce extreme suffering. (authors emphasis)

Suffering is defined thus:

“By suffering and happiness I refer to an overall state of feeling in a given moment, with suffering being an overall bad feeling, or state of consciousness, and happiness being an overall good feeling” (pg.13)

One common criticism, which Vinding gracefully counters early on, is that negative experiences such as pain are not always bad (for more see “Pain: The Gift Nobody Wants”).

“For example, one’s experience may contain a component of pain or bittersweet sorrow, but if this component does not render the overall experience negative or disagreeable, then this experience does not constitute suffering on the definition employed here”. (ibid)

The first three chapters serve as a compendium of sorts, detailing views from a multitude of authors defending the first two of Vinding’s four theses. At least one reviewer has felt alienated by this approach, saying that “many of the arguments are of the form "philosopher X thinks that Y is true", but without appropriate arguments for Y”, adding that, “I was being persuaded, not explained to”.

If these chapters are viewed as a collection of self-contained arguments, then it is easy to see how the reviewer could have felt that way. However, if these chapters are understood as merely a collection of views held by thinkers in the past, to be used as a resource for those interested in the field, then the criticism dissolves. Certainly however, Vinding could have done more to emphasize that his goal in these chapters was not to reason from first principles, as he does more in the next few chapters, but merely to showcase what views there were.

I will now analyse the views and arguments in the following two chapters: chapter four, and chapter five. These two chapters, out of all eight chapters in section one,  contain the bulk of Vinding’s original arguments. In chapter four, Vinding grants us with a new phrase, the ‘Principle of Sympathy for Intense Suffering’ (henceforth, the Principle). In chapter five, he will discuss and defend a position on Moral Realism.

Principle of Sympathy for Intense Suffering

“What this principle says, roughly speaking, is that we should prioritize the interests of those who experience the most extreme forms of suffering. In particular, we should prioritize their interest in not experiencing such suffering higher than we should prioritize anything else.” (pg.60, my emphasis)

The first argument Vinding furnishes us for the Principle takes the form of a rhetorical question:

“Something that may help motivate us to take this principle seriously is to ask, quite simply, why we would suppose it to be otherwise. Why should we believe that the most extreme forms of suffering can somehow be outweighed or counterbalanced by something else? After all, there seems no reason, a priori, to suppose this to be the case.” (ibid)

In general, I agree with the Principle. However I’m not sure whether it is for the same a priori reasons. When I first read the Principle, I immediately thought of a few other overriding values where it prima facie (meaning "at first sight") appears that “the most extreme forms of suffering can somehow be outweighed or counterbalanced by something else”. I will list two of these values now, and then discuss how Vinding might respond.

One such value is love. A certain romantic notion of love consists of people pursuing their love interest in the face of overwhelming odds and risks. One may proclaim to their lover that they would be willing to go to the depths of hell and back for the sake of the other. Another value is comradery in battle. A soldier charging into battle may care about the livelihoods of his brothers in arms more than he may care about any suffering he will bring about. Soldiers may maim, rape, and torture the enemy in war with little regard to suffering whilst in the midst of it. In both cases, concerns for suffering are prima facie being overridden by other values.

Vinding may respond with the following (replacing the word ‘pleasure’ with ‘comradery/love’):

“After all, common sense would seem to say that if a conscious subject considers some state of suffering worth experiencing in order to attain some given pleasure, then this pleasure is indeed worth the suffering. And this view may work for most of us most of the time. Yet it runs into serious problems … in cases where subjects consider their suffering unoutweighable by any amount of pleasure.” (pg.61)

“[This would be because pleasure cannot outweigh] suffering so intense [since] they 1) consider it unbearable — i.e. they cannot consent to it even if they try their hardest — and 2) consider it unoutweighable by any positive good, even if only for a brief experience-moment.” (pg.62)

Thus, even though a lover or a soldier may claim that they are willing to undergo or cause suffering for the sake of certain values, that is love or comradery, they will not consent to this view the moment they face real suffering. They will want the suffering to stop, no matter how hard they try to place comradery/love on the pedestal, and will consider suffering “unoutweighable“ by these other values.

This makes sense to me, and my reaction is to say that yes that is true, the lover and the soldier will forfeit their prior claims in the face of genuine suffering. However, having said that, I still cannot help but feel that “surely the power of love will overcome any obstacle no matter how tough”. But that might just be a view which watching old disney movies has instilled within me. “Love overcomes any obstacle”, I tell myself, scrolling through tinder. I will return to these niggling thoughts on the difference between having a feeling, and what these feelings actually imply, in the “is/ought” section in my review of the next chapter.

The remaining arguments for the Principle consist of appeals to: 1) common-sense, and 2) the horrors of the world.

Common-Sense Argument

Of common-sense, Vinding uses a roller coaster analogy:

“[The Principle] has a lot of support from common sense. For example, imagine two children are offered to ride a roller coaster — one child would find the ride very pleasant, while the other would find it very unpleasant. And imagine, furthermore, that the only two options available are that they either both ride or neither of them ride (and if neither of them ride, they are both perfectly fine). Which of these options should we ideally realize?“

“Common sense would seem to say we should sympathize with and prioritize the child who would find the ride very unpleasant, and hence choose the outcome in which there is no harm and no victim.” (pg.63)

I suspect there is a lot being loaded onto the words “common sense” here. Pushing past that, Vinding successfully expands on the scenario in a few ways, and justifies his reasons in both. He i) expands the thought experiment by including multiple children being benefited and suggests that these multiple benefiters still cannot outweigh the harm of the one child, and he ii) compares the roller-coaster situation - which does not include extreme suffering - to one that includes extreme suffering, and says that having extreme suffering involved would make ignoring the child receiving the harm even more repulsive.

Horror Argument

Of horror, he quotes a panoply of authors, tortured souls, forms of execution and devices of human suffering. Take the example of sawing:

“We can begin to approach such appreciation by considering what it is like to undergo “death by sawing”, an execution method used on humans in many parts of the world in the past, and which is still used on non-human animals in some parts of the world today. The victim is tied up and then sawn or chopped in half with a saw or a machete, either transversely or midsagittally — in the latter case, it has both been done from the groin up as well as from the skull down.” (pg.65)

The idea is that there is immeasurable suffering in the world which we do not truly appreciate, sitting in our comfy couches reading book reviews online. As crude as such an argument is, I believe these arguments nevertheless serve as the strongest proponents for the Principle, if just for the sheer shock value. This is even though they do not rely on straightforward syllogistically reasoning but an appeal to emotion, so it is good that Vinding does not shy away from using these arguments but rather embraces them.

Overall I am convinced that the Principle is true. Where does Vinding go next, having established it? He will now try to show not only that the Principle true, but that it undergirds our moral reality:

Moral Realism

“A Moral Realist Case for Minimizing Extreme Suffering” is the most out-of-place chapter in ‘Suffering-focused Ethics’.

Vinding begins:

“In this chapter, I shall make a moral realist case for the view that we ought to minimize extreme suffering. In other words, I will argue that we in a sense truly and objectively ought to reduce extreme suffering as much as we can.” (pg.75, author’s emphasis)

Typically, discussions of moral realism are meta-ethical discussions. That is, a discussion about ethics, rather than ethical discussion itself. When discussing more practical concerns, (relatively practical, anyway) such as why we should alleviate suffering, meta-ethical concerns typically don’t have a part to play. One can usually believe the positions outlined so far in Vinding’s book, independent of one’s meta-ethical commitments. (Worth noting too, is that Moral realism, while the most popular view in philosophical circles, is certainly not the view of the overwhelming majority: 27.7% of philosophers are moral antirealists, according to a famous Philpapers survey.)

For context, moral realism is a notoriously overloaded word (if there ever was a consensus of understanding about ‘realism’, as a philosophical term of art, it has undoubtedly been fragmented by the pressures exerted by the various debates—so much so that a philosopher who asserts that she is a realist about theoretical science, for example, or ethics, has probably, for most philosophical audiences, accomplished little more than to clear her throat.” Crispin Wright, in the SEP article on anti-realism)  which, depending on who you ask, is a mix of moral cognitivism (the claim that moral claims can be true or false), moral objectivism (the claim that moral claims are not indexed on a specific person or culture), and possibly moral absolutism (that some moral claims can override any other, and always apply). Which subset of the above claims moral realism consists of, are further muddied by views such as Moral Error Theory, which say that moral claims can be true or false, but are nevertheless always false. (As an example, consider “Unicorns have a single horn”, or “God is good”, which a Fiction Error Theorist or a Religious Error Theorist may say can be informative, interesting, and even forceful claims, but are nevertheless trivially false because Unicorns/Gods do not exist.)

Of-course, moral realism can be a meaningful phrase, and academics use it with vigour, but I bring up this definitional quandary because we will soon see that Vinding will encounter troubles with the phrase.

So, having said all that, my first reaction upon seeing this chapter was: “Vinding! By arguing for a “moral realist case” are you trying to answer this famous long-standing philosophical dispute on moral realism in a single chapter, and furthermore show that moral realism must align with reducing suffering?” To my mind, both claims are book-worthy in themselves. Therefore, even though I find Vinding’s arguments in this chapter to be mostly lacking, I can nevertheless praise Vinding for his aspirations.
(LAST MINUTE EDIT: Just before submitting this review, I’ve been informed that Magnus wrote a book exploring his meta-ethical stances. I have not read it, but will hyperlink it for those interested.).

Vinding says his views rest on two premises:

“My moral realist argument for minimizing extreme suffering rests on two premises. The first premise is that suffering is intrinsically disvaluable and carries normative force. In particular, any subject experiencing a state of extreme suffering will be forced to conclude that this state, by its very nature, is truly disvaluable and that it truly ought not be. The second premise is that the disvalue and normative force of extreme suffering is qualitatively greater than anything else, and hence the extent to which our actions minimize such suffering is the supremely most important thing about them. In sum: minimizing extreme suffering carries genuine and supreme normative force.” (pg.87, author’s emphasis)

And weakens the argumentative leverage for such views shortly after:

“These premises are difficult to argue for, as opposed to just blankly state. And although I will attempt to justify these statements in the following sections, I think their truth can ultimately only be verified through direct experience.”

Before tackling his “moral realist argument”,  let’s understand what Vinding himself means by the term ‘moral realism’.

Vinding’s Definition of Moral Realism

Vinding does not explicitly define moral realism, however he does brush against the edge of such a definition:

“The view I present here can be seen as a moral realist version of the principle argued for in the previous chapter, as it defends a version of [The Principle] as a moral truth” (pg.75, my emphasis)

Therefore, whatever moral realism is for Vinding appears to be a form of moral cognitivism: the view that moral claims can be true or false. Moreover, Vinding believes that only if one is a moral realist, can one understand suffering to be truly disvaluable, that it truly ought not to be, and that is has genuine and supreme normative force.

Thus, from what we gather above, the implication is that if one were not a moral realist, one would not feel the supreme normative need to end suffering, and would not find it truly disvaluable. That is, perhaps one may find suffering somewhat disvaluable, and one may feel only a slight normative force towards ending suffering. Having said that however, I find it quite implausible that 27.7% of academic philosophers have no genuine desire to end suffering, and hold perhaps only a quasi-, semi-, fake, or mild form of desire. Therefore, we can conclude that were Vinding to have a coherent definition of moral realism which does not implausible dismiss the desires of a quarter of philosophers (even though that may not be his intention), it must be different from the mainstream definition(s) used in academia.

Since Vinding does not hold a mainstream view of moral realism, he may lay claim to a consistent definition of moral realism in one of three ways: 1) Either he has to redefine moral realism as the phrase denoting the view which holds suffering as truly disvaluable (which is question begging), or 2) he must admit that the use of the words truly, genuine, and supreme here have no actual import and are being used purely as rhetorical devices (which would be both unsatisfactory, and potentially embarrassing), or lastly he can 3) retreat to a private language view, whereupon his views would not only be “difficult to argue for” and ultimately “verified through direct experience”, but would in fact be impossible to argue for, for he would have his own peculiar view on “moral realism” unknowable to the reader.

Based on the three routes above, it is clear that Vinding cannot hold a view of moral realism which is not question-begging, which is not shallow in its meaning, and which does not make the following arguments impossible to interpret.  Nevertheless, disregarding all of this, let's assume quite plausibly that I have missed something, and that Vinding indeed has a coherent account towards a “moral realist case for minimizing extreme suffering” which is worth arguing for in this book. What are his arguments for it?

Arguments for Moral Realism

Vinding claims that for his case to hold, he must prove two premises for moral realism as it relates to ending extreme suffering:

1. That suffering is inherently disvaluable,
2. That extreme suffering has supreme normative force

I will tackle these in turn.

1 - Suffering is Inherently Disvaluable (5.4)

In this section, Vinding discusses views from other thinkers (Richard Ryder, Henry Sidgwick, Jamie Mayerfeld, and Karl Popper), and then presents his own view:

“On my account, this is simply a fact about consciousness: the experience of suffering is inherently bad, and this badness carries normative force — it carries a property of this ought not occur that is all too clear, undeniable even, in the moment one experiences it. We do not derive this property. We experience it directly.” (pg.79)

And then whips us into attention with two absolutely incredible paragraphs, which I will quote in full:

“In my view, this is what bridges the “is-ought gap” that is said to prevent us from deriving statements about how the world ought to be from statements about how the world is. The statement “suffering is, by its nature, something that ought not occur” is a statement about how the world is, and it does say something about how the world ought to be. We are only forced to accept a gap between “is” and “ought” if we accept the premise that there are no states of the world — no states of what is— that have intrinsically normative properties, a premise I firmly reject.”

“By analogy, consider mathematical lines and vectors. Saying that “ought” cannot be found anywhere in the realm of what is, is akin to saying that there are only lines, whereas vectors, things that point in some direction, do not exist. I disagree. When we look closer, we find that vectors — mathematical as well as ethical ones — indeed do exist in a very real sense (though this is not to say they exist in any Platonic sense). The world is richer than our ordinary narrow conceptions suggest.” (pg.80)

There’s a lot to be said here.

Firstly, the is-ought problem (that you cannot derive an ought, using just facts about the world) is far from being a closed question in ethics and while this paragraph has highlighted one path out of it, here’s one counter-argument against it:

Just because an experience has an attached drive (or in Vinding’s words, a “normative force”) does not mean that it suddenly becomes an ought. For example, if a man cuts me in a queue, I would feel tempted to smack him on the head. This would be “all too clear, undeniable even, in the moment one experiences it”, yet this is not what I ought to do. There are overriding factors, such as my understanding that it’s not morally acceptable. Likewise, if after I’ve calmed down I see an alluring woman dressed in red I might want to approach her and feel a desire to give her a great, big, hug. Yet, this is not what I ought to do.

Similarly, if I go through several sleepless nights and cry out I would rather my whole world vanish around me than spend another hour in sleepless agony, that does not mean it ought to end. Such an experience could be overridden by other values, such as comradery  - “guarding my comrades from the enemy at nightwatch, is worth these hours of torment” - or if not, then could be re-evaluated in the future - “I am glad I went through that ordeal, because it has made me appreciate a good night’s sleep”.

Granted, if it’s neither overridden, nor re-evaluated, I admit that crying out to stop the suffering from sleepless nights will have normative force simplicitur. But this is not unique to suffering: If anger is not overridden by morality, then it too has normative force simplicitur - simply my hand towards the fellow’s head - Smack!

Of-course, Vinding may reply that I have not truly experienced or fairly represented real suffering if my response is to be lackadaisical and not to want to end the suffering regardless of my other values, but such a response would feel to me like a type of No True Scotsman argument: any form of suffering that does not entail “ought not to occur” is not really a form of suffering, for ought-ness is the defining property of suffering. He may be right, granted he says that direct experience is the ultimate judge of his position, but if that’s the case, then it is not a topic worth discussing for we would always speak past each other.

Secondly, the lines and vectors analogy is the strangest paragraph of the entire book.

In a universe consisting of lines, “when we look closer, we find that vectors — mathematical as well as ethical ones — indeed do exist in a very real sense”. I do not know what to make of this. Whether mathematically lines exist in reality is a whole topic in itself, and whether vectors follow from lines is not trivially true at all, for you must add a sense of direction to a line, in order to make it a vector. Also, what is an ethical vector, and how does it compare to ethical lines? Further, why is discussion of is and ought akin to lines and vectors? This paragraph appears to be several layers removed from any meaningful interpretation, so perhaps this is just a case of a wide inferential distance between Vinding’s mind and mine.

2- Extreme Suffering has supreme normative force (5.5)

Vinding argues for this in two steps.

“A first step may be to argue that all moral value ultimately resides in the realm of consciousness: something can ultimately only have moral value if it affects the experience of a sentient being.”

“If we grant this step, all we would need to do … is to establish that extreme suffering has superior moral significance relative to all other states within the realm of consciousness.” (pg.85)

Vinding is careful to temper this position with what he calls throughout his book a “pluralism” about morality: that the prevention of Suffering, while overridingly important, is not the only good of value.

“It is important to note that the view that extreme suffering has overriding normative force is by no means predicated on the view that all moral value is found within the realm of consciousness. For instance, one can believe that things such as knowledge, art, and virtues have intrinsic moral value not reducible to their effect on conscious subjects while still believing that the reduction of extreme suffering has supreme importance.”

Coming back to his argument, we see that it is a logical two step procedure: 1) Show that morality value resides in consciousness, and 2) show that extreme suffering must have “superior normative significance” compared to other states of consciousness. He argues for the first step using a convincing planet analogy - that we would rather reduce suffering on planet A, than place an intrinsically valuable non-experiential good (whatever that may be) on planet B. His argument for the second is shakier.

To show that second step, that extreme suffering has more moral significance than other states of consciousness, Vinding must establish extreme suffering’s “superior importance relative to mild suffering” (pg.88). He does so by claiming that mild suffering is lexically different to extreme suffering. He defines lexical thus:

“a single instance of such extreme suffering is worse and more important to prevent than arbitrarily many instances of such mild suffering” (pg.88)

How he shows there is a lexical difference is by analogy to physical phenomena. In a wide range of physical phenomena, a small difference can cause a sudden change in state. Therefore, as some transition states of physical phenomena do not admit of grey areas, so too do some transition states of suffering not admit of just grey areas. A discontinuous jump implies a lexical difference:

“For example, a spring will exert a pull with a certain force depending on how far we stretch it, yet at a certain point, if we pull the spring just a tiny bit further, the spring will break and its pull will go abruptly to zero. Likewise, consider the discrete energy levels of atoms, where just a tiny difference in how much energy we bombard an atom with can mean a qualitative difference — a literal quantum leap — between two discretely different states with nothing in between.” (pg.91)

Yet, one problem with this analogy to physical phenomena is that each phenomena has an understandable mechanism. Atoms do quantum leaps because we have an understanding of quantum physics, and springs fail to deform elastically (or break) beyond a certain point because of our understanding of yield stresses and chemical bonds.

To my knowledge, there is no analogous mechanism in the brain for suffering. Perhaps Vinding may argue that there are certain neural correlates of suffering in the brain that require minute activations in order to transform a mild type of suffering to an extreme one. If this is so, I have never heard of such a correlation, and would be interested in knowing more.

The above’s relation to moral realism

Overall, while I am convinced by his conclusions, I am not convinced by Vinding’s arguments that 1) suffering is inherently disvaluable, and 2) that extreme suffering is supremely significant compared to other states of consciousness, let alone that moral realism inherently relates to either of these views.

What would drive Vinding to frame these positions through the lens of moral realism?

One “potentially motivating force” he provides for pushing this agenda is that there is a psychological reason to believe in moral realism:

“... it seems natural to expect that moral realism is better able to motivate moral behavior. And this conjecture finds support in empirical studies. For example, one study in which participants were primed toward either moral realism or anti-realism found that:

Participants primed with realism were twice as likely to be donors [to charity], compared to control participants and participants primed with antirealism. … [In sum:] priming a belief in moral realism improved moral behavior”
(pg.77, quoted in Vinding. Link to study quoted:  https://www.sciencedirect.com/science/article/abs/pii/S0022103112002375 :Young & Durwin, 2013)

Yet, the question of whether X is true is independent of whether it is worth believing X. Some argue that it is worth believing in religion, even if there may be no God. Moral Error Theorists (such as Joyce), hold views that ethics is a “useful fiction”, even if ethical beliefs themselves have no truth-seeking property. Thus, while Vinding has taken a pragmatic approach by trying to show that we should believe in a moral realist account of his views, it is nevertheless independent of whether such a view is actually true.

Vinding partially agrees:

“Yet it should be noted that I do not believe in moral realism because believing it seems to have good consequences. Rather, it is in a sense the other way around: it is because I think there is such a thing as truly better or worse consequences in the first place that I am a moral realist. The potential motivating force of realist views just gives me additional reason to try to convey my view.” (pg.77, my emphasis)

However I worry, from a purely philosophical standpoint, how efficacious (can it backfire?) and honorable it is to nudge people towards a view that may be a useful “motivating force” and driver of ethical action, if it nevertheless has no truth baked into it. Having said that I understand where Vinding is coming from, so I will leave this discussion for another day.

Section 2 - How Can We Best Reduce Suffering?Chapters in Section 2

In this shorter, second section of the book Vinding deals with a whole swath of topics. This section can be viewed as a handbook of sorts, for the now-convinced suffering-focused Vindingian-utilitarian. The topics include but are not limited to:

Moral Uncertainty, Cooperation, Non-Human Animals, The Abolitionist Project, S-risks, and Self-Investment.

I will be focusing on his views on S-Risk for the rest of the review, as this is the topic that has interested me the most. If, however, you take a fancy to any of the other topics listed above, and they are worth reading, you are encouraged to have a read of them here.

S-risks

“It is conceivable that future space colonization will create orders of magnitude more suffering than anything that could ever be contained on Earth. The risk that suffering might be realized on such a large scale has been called a “suffering risk” or “s-risk”, and reducing such risks should be a priority on all views concerned about future suffering.” (pg.247, introduction to Chapter 14: “Reducing S-Risks”)

Space Colonization plays a large role in Chapter 14, a chapter dedicated to S-Risk. In fact, it plays such an overwhelming role in the chapter that I am surprised Vinding did not just name the chapter “Space Colonization”. In my review of this section I will outline Vinding’s views on why Space Colonization is harmful. I will agree with his views for the most part, however I will show that he did not go far enough in his analysis, for he neglected the impact which life outside earth (were it to exist) would have on his expected value calculations.

For Vinding, in order for an S-Risk to occur we only need to factor in two considerations, that:

“1) large-scale space colonization will happen, and 2) [that] this colonization will involve an insufficient concern for suffering. This combination of conditions can hardly be written off as having extremely low probability, especially given that serious projects aimed to colonize Mars already exist, and that humanity currently displays a gross lack of concern for the suffering of sentient beings“ (pg.248)

Vinding furthermore cites Althaus/Gloor in calling S-Risks “disjunctive - they can be realized in many different ways that bear little relation to each other”.  Due to this disjunctivity - that we may cause an S-Risk through wide scale factory farming, proliferation of interstellar war, S-Risks due to AI risk, e.t.c. - and by virtue of the fact that we are psychologically known to undervalue disjunctive risk, we should pay more heed to possible S-Risks, even if they may not be on our immediate radar.

I find the space colonization argument prima facie convincing, and the disjunctive argument is also convincing, for it says we’re biased, after all. Knowing this, what does Vinding propose we should do in the face of possible S-Risks?

Reduce possible Dystopias

One step to reducing S-Risks is to aim towards prevention of dystopias, which is more tractable than aiming for utopia creation.

“Yet not only does s-risk reduction seem highly neglected, it also seems more tractable than comparable large-scale objectives, such as utopia creation. Put simply: the asymmetry in ease of realization suggests that it is more cost-effective (for a wide variety of value systems) to work to avoid dystopian outcomes than to create utopian ones.” (pg.254)

And as evidence Vinding quotes Althaus/Gloor again, and also refers us back to Section 1.3, which claims that there is an asymmetry between bringing about suffering and bringing about pleasure. That is, “it is much easier to bring about states of extreme suffering than to bring about states of extreme happiness”, and this asymmetry is true “in various levels for various reasons”: “failure is generally much easier than success. For example, there are many more ways not to solve a Rubik’s Cube than to solve it.” Vinding also discusses this in the form of the “Anna Karenina Principle”.

Yet, as understandable as his metaphors are, I think Vinding has moved a bit too fast here. There are more than a few steps needed to go from 1) the view that realising suffering and realising happiness are asymmetrical, to 2) the claim it is more “cost-effective to work to avoid dystopian outcomes than to create utopian ones.”

Firstly, I question whether striving to create a utopian outcome is, in reality, a driving force for any government or high-impact individual. That is, most charities appear to be aiming to alleviate suffering in some form, rather than aiming for creating utopias in the abstract (with the exception, perhaps, being art museum charities). So, even if Vinding is right, and given it’s the case that few appear to be aiming for utopic outcomes as such, is he in some sense just arguing against a blank wall?

Secondly, if it’s really the case that it’s easier to tend towards disarray and dystopia than it is to tend towards utopia - like how a Rubiks cube has more failure states than success states - how would he explain the fact that we are not, for all intents and purposes, living in a dystopian nightmare right now? I accept there have been a few disastrous semi-dystopic hiccups in humanities collective path through the universe: the various wars and genocides, the poor air quality and child labour during the industrial revolution, perhaps the agricultural revolution, and recently the fake-news culture wars, but in return we get modern medicine, more time than ever to pursue our hobbies and passions,  and of-course the (historically) recent dissemination of knowledge to the masses through the invention of books and the internet!

I can agree that realising suffering and realising happiness are asymmetrical endeavors where one appears to be easier to achieve than the other in a sense, but in another sense (whatever that sense may be, I don’t know myself) it’s also a fact that both individually and collectively we are heading towards a better society, and not away from it. I would have liked to see Vinding clarify his position here, and expand more on his thought that “failure is generally much easier than success”, and its implication for his view that it’s better to “work to avoid dystopian outcomes than to create utopian ones”.

With the exception of the above claim regarding utopia, his other steps towards reducing S-Risks are much more balanced. That is, we should focus our attentions on “robust” (Vinding does not define ‘Robust’ explicitly, but I presume it means something along the lines of “applicable regardless of the particular S-Risk strategy you want to minimize”) strategies such as i) building up the sentiment that we should reduce S-Risks, ii) fostering cooperation, and iii) advancing S-Risk research by “think[ing] through many strange possibilities while avoiding the trap of unwarranted confidence in speculative hypotheses”

Vinding admits that these not sexy endeavours:

“Such a mundane mission of capacity-building, cooperation, and research-promotion hardly seems like a particularly attractive thing to be affiliated with for our status-obsessed brains.” (pg.255)

But nevertheless says that in the pursuit of preventing the greatest amount of horrors we could possibly face, considering just sexy, status-inducing options is not the way to go.

So to help Vinding on the project of finding “strange possibilities” of S-Risk, while possibly being overconfident in my own “speculative hypothesis”, I present the following addition to the discussion on space colonization:

Aliens

It is known that aliens could exist in the universe. Furthermore, it is also known that aliens could have the following properties (ordered by increasing order of complexity): 1) the ability to suffer 2) the ability to act intelligently, 3) the ability to create their own S-Risks, and 4) the ability to expand into the universe propagating their own S-Risks.

Let’s further assume that we care about S-Risks, and we care to a very large degree about “orders of magnitude more suffering” compared with what “could ever be contained on Earth.”

Building upon any of the 4 possibilities above, multiplied by how many aliens we expect to exist in the universe which will be reachable by us in the future by robotic spacecraft or whatever means we have, we see that we have a clear moral imperative for space colonization in order to prevent any of these disastrous scenarios from happen, or continuing to happen.

Firstly if aliens could suffer, even if they were to exist as mere balls of goo rummaging on an alien ocean, grinding the soil for nutrients and tearing each other apart for a slice of the alien pie, then given that Vinding cares about Wild Animal Suffering (More in Chapter 14), he must therefore care to a higher degree about all the Wild Animal Suffering that could be prevented cosmically. Whatever suffering animals exist now, or will in future exist on earth, are insignificant compared to what exists or will exist cosmically within our means of prevention.

Secondly, if aliens could act intelligently, they may cause widespread destruction towards their own ecosystem. They will have their own world wars, moral disasters, and genocides. Instead of humans they may be ‘Schmoomans’, similar to us and causing all the similar evils and problems which we ourselves cause to ourselves, except that they exist far away from us. Whatever Vinding thinks of harm caused by humanity must be multiplied by magnitudes more given all the schoomans that will exist, even considering just the Schmoomans in our galaxy, let alone the universe.

Thirdly, if they can create their own S-Risk, it is possible that right now as you are reading this, there are several thousand super-intelligent AIs in the universe that are currently holding hostage googly-eyed aliens and churning their bodies into paperclips. They will be stuck in such an equilibrium of torment unless an external force, such as humanity (or our own, benevolent AI) comes and rescues them. As unlikely as any of this may be, it is still a possibility that we must consider with earnestness, given that calculations of S-Risks as understood by Vinding (i.e. without aliens) already includes low probabilities with high uncertainties.

Lastly, the aliens could create/have created/will create their own S-Risk, and be propagating the universe with this S-Risk, just as Vinding envisages we may to ourselves if we colonize space. However, either 1) this is already happening/will happen, in which case we ourselves will be subsumed by them in the millions of years to come, or more likely we are an early civilization in the universe (according to Robin Hanson’s “Grabby Aliens” model) so, 2) quite possibly there are no grabby aliens populating the universe with S-Risks yet, so we have the very special, once in a universe opportunity to be the force of benevolence in the universe, propagating space with hope, peace, happiness, and utopia.

Vinding may argue that as convincing as he finds the above arguments, it is still way too risky to throw our net into the universe, knowing that we have such a high chance of propagating an S-Risk onto ourselves. My response is that whatever S-Risk we create towards humanity (or sentient terrestrial life) will very likely be a drop in the ocean in the literal universe of suffering that exists around us, or will exist around us. Be it through the base case of aliens which can suffer, or the more extreme case of Schmoomans which will consume the universe with S-Risk. Both are undesirable scenarios which we have a chance to prevent. So, even if humanity becomes a part of a global paperclip churning machine (or whatever disastrous S-Risk you envision), it is a risk worth taking to prevent the literal hundreds of millions of paperclip churning machines that could exist now, or in the future.

Accepting Vinding’s idea that Extreme Suffering cannot be overridden by pleasures, but have an intrinsic and overriding moral worth in preventing by themselves, Vinding must recognise that this is a battle between

1. A (possibly temporary) peaceful life on earth, with galactic scale suffering elsewhere - if we do not colonize space, or

2. Galactic scale suffering everywhere, with this small chance of galactic scale suffering being eliminated for all - if we colonize space (with that small chance succeeding if colonize space successfully).

Knowing that humans have no claim in being any more valuable than Schmoomans, for we are both creatures of suffering, it is clear to me which option I prefer!

Furthermore Vinding should welcome these arguments, for I am suggesting a form of optimism: unlike most future civilizations who will reach some equilibrium state (either by reaching extinction, or by being subsumed by an external civilization at some unspecified point in the future), as an early civilization we are uniquely situated at the crossroads of preventing large amounts of suffering in the universe. While the road will be bumpy and tiresome, and there will be plenty of room for error, it’s a shot humanity’s been given, and it’s one we must use.

Conclusion

In my review of the first section of “Suffering-focused Ethics” by Magnus Vinding I have summarised his main points, and dived into a deep discussion of both his “Principle of Sympathy for Intense Suffering” and his “Moral Realist case for Suffering-focused Ethics”. I have shown that his arguments in both cases, while generally unconvincing, do contribute to the growing literature on the topic and his chapters here may act as a reference point for those who are new to the field.

In my review of the second section, I have analysed his views against Space Colonization and S-Risk, finding his arguments somewhat convincing, but definitely praiseworthy by placing into the spotlight a rarely discussed position with regards to Space Colonization. Certainly, he has given me something to ponder on. In my review I also have contributed to the discussion, by highlighting how the introduction of suffering-able aliens radically change his fundamental arguments against Space Colonization.

Overall I do not find Vinding a persuasive arguer. However, I do find him to be one of the few individuals spearheading the movement for suffering-focused ethics, so while some of his arguments are lacking, his positions are worth serious consideration. While I did not tackle every chapter of the book (given more time and energy I would have reviewed his views on “The Abolitionist Project” as well as “Non-Human Animals”), I hope I’ve given the reader enough material for them to decide whether or not this work is reading for themselves.

I will be looking forward to reading future works of Vindings. Magnus Vinding is one of the co-founders of the Center for Reducing Suffering.

Thanks

Many thanks to those friends of mine who provided feedback (and wish to remain anonymous). Thanks to the LessWrong team for their book review contest, which helped motivate me to write this essay.

The review is also available on my blog.

Discuss

### In the shadow of the Great War

19 октября, 2021 - 02:08
Published on October 18, 2021 11:08 PM GMT

The idea of progress fell out of favor in the course of the 20th century. But when exactly, and why?

In a recent essay I alluded to the pivotal role of the World Wars. Here’s a quote that adds weight to this—from Progress and Power, by historian Carl Becker, published in 1936:

For two centuries the Western world has been sustained by a profound belief in the doctrine of progress. Although God the Father had withdrawn into the places where Absolute Being dwells, it was still possible to maintain that the Idea or the Dialectic or Natural Law, functioning through the conscious purposes or the unconscious activities of men, could be counted on to safeguard mankind against future hazards. However formulated, with whatever apparatus of philosophic or scientific terminology defended, the doctrine was in essence an emotional conviction, a species of religion—a religion which, according to Professor [J. B.] Bury, served as a substitute for the declining faith in the Christian doctrine of salvation …

Since 1918 this hope has perceptibly faded. Standing within the deep shadow of the Great War, it is difficult to recover the nineteenth-century faith either in the fact or the doctrine of progress. The suggestion casually thrown out some years ago by Santayana, that “civilization is perhaps approaching one of those long winters which overtake it from time to time,” seems less perverse now than when it was made. Current events lend credit to the prophets of disaster who predict the collapse of a civilization that seemed but yesterday a permanent conquest of human reason …

At the present moment the world seems indeed out of joint, and it is difficult to believe with any conviction that a power not ourselves—the Idea or the Dialectic or Natural Law—will ever set it right. The present moment, therefore, when the fact of progress is disputed and the doctrine discredited, seems to me a proper time to raise the question: What, if anything, may be said on behalf of the human race? May we still, in whatever different fashion, believe in the progress of mankind?

I find it fascinating to see that the downfall of the idea of progress began as early as this, after World War I. World War II perhaps simply reinforced an existing trend.

I also find fascinating Becker’s idea that humanity required some sort of safeguard, a “power not ourselves” to “set it right.”

There is no power outside of humanity. We are the masters of our fate, for better or for worse. If there is to be a 21st-century philosophy of progress, it needs to be based not on an Idea or a Dialectic, but on human agency.

Discuss

### Prioritization Research for Advancing Wisdom and Intelligence

19 октября, 2021 - 01:28
Published on October 18, 2021 10:28 PM GMT

LessWrong note: I wrote this more in a way slightly more optimized for the EA Forum than LessWrong, because the post seemed slightly more appropriate there.

Summary

I think it makes sense for Effective Altruists to pursue prioritization research to figure out how best to improve the wisdom and intelligence[1] of humanity. I describe endeavors that would optimize for longtermism, though similar research efforts could make sense for other worldviews.

The Basic Argument

For those interested in increasing humanity’s long-term wisdom and intelligence[1], several types of wildly different interventions are options on the table. For example, we could improve at teaching rationality, or we could make progress on online education. We could make forecasting systems and data platforms. We might even consider something more radical, like brain-computer interfaces or highly advanced pre-AGI AI systems.

These interventions share many of the same benefits. If we figure out ways to remove people’s cognitive biases, causing them to make better political decisions, that would be similar to the impact of forecasting systems on their political decisions. It seems natural to attempt to figure out how to compare these. We wouldn’t want to invest a lot of resources into one field, to realize 10 years later that we could have spent them better in another. This prioritization is pressing because Effective Altruists are currently scaling up work in several relevant areas (rationality, forecasting, institutional decision making) but mostly ignoring others (brain-computer interfaces, fundamental internet improvements).

The point of this diagram is that all of the various interventions on the left could contribute to helping humanity gain wisdom and intelligence. Different interventions produce other specific benefits as well, but these are more idiosyncratic in comparison. The benefits that come via the intermediate node of wisdom and intelligence can be directly compared between interventions.

In addition to caring about prioritization between cause areas, we should also care about estimating the importance of wisdom and intelligence work as a whole. Estimating the importance of wisdom and intelligence gains is crucial for multiple interventions, so it doesn’t make much sense to ask each intervention’s research base to independently tackle this question on their own. Previously I’ve done a lot of thinking about this as part of my work to estimate the value of my own work on forecasting. It felt a bit silly to have to answer this bigger question about wisdom and intelligence, like the bigger question was far outside actual forecasting research.

I think we should consider doing serious prioritization research around wisdom and intelligence for longtermist reasons.[2] This work could both inform us of the cost-effectiveness of all of the available options as a whole, and help us compare directly between different options.

Strong prioritization research between different interventions around wisdom and intelligence might at first seem daunting. There are so clearly many uncertainties and required judgment calls. We don’t even have any good ways of measuring wisdom and intelligence at this point.

However, I think the Effective Altruist and Rationalist communities would prove up to the challenge. GiveWell’s early work drew skepticism for similar reasons.  It took a long time for Quality-Adjusted Life Years to be accepted and adopted, but there’s since been a lot of innovative and educational progress. Now our communities have the experience of hundreds of research person-years of prioritization work. We have at least a dozen domain-specific prioritization projects[3]. Maybe prioritization work in wisdom and intelligence isn’t far off.

List of Potential Interventions

I brainstormed an early list of potential interventions with examples of existing work. I think all of these could be viable candidates for substantial investment.

• Human/organizational
• Rationality-related research, marketing, and community building (CFAR, Astral Codex Ten, LessWrong, Julia Galef, Clearer Thinking)
• Institutional decision making
• Academic work in philosophy and cognitive science (GPI, FHI)
• Cognitive bias research (Kahneman and Tversky)
• Research management and research environments (for example, understanding what made Bell Labs work)
• Cultural/political
• Freedom of speech, protections for journalists
• Liberalism (John Locke, Voltaire, many other intellectuals)
• Epistemic Security (CSER)
• Epistemic Institutions
• Software/quantitative
• Positive uses of AI for research, pre-AGI (Ought)
• Tools for thought” (note-taking, scientific software, collaboration)
• Forecasting platforms (Metaculus, select Rethink Priorities research)
• Data infrastructure & analysis (Faunalytics, IDInsight)
• Fundamental improvements in the internet / cryptocurrency
• Education innovations (MOOCs, YouTube, e-books)
• Hardware/medical
• Lifehacking/biomedical (nootropics, antidepressants, air quality improvements, light therapy, quantified self)
• Genetic modifications (Cloning, Embryo selection)
• Digital people (FHI, Age of Em)
Key Claims

To summarize and clarify, here are a few claims that I believe. I’d appreciate insightful pushback for those who are skeptical of any.

1. “Wisdom and intelligence” (or something very similar) is a meaningful and helpful category.
2. Prioritization research can meaningfully compare different wisdom and intelligence interventions.
3. Wisdom and intelligence prioritization research is likely tractable, though challenging. It’s not dramatically more difficult than global health or existential risk prioritization.
4. Little of this prioritization work has been done so far, especially publicly.
5. Wisdom and intelligence interventions are promising enough to justify significant work in prioritization.
Open Questions

This post is short, and of course, leaves open a bunch of questions. For example,

1. Does “wisdom and intelligence” really represent a tractable idea to organize prioritization research around? What other options might be superior?
2. Would wisdom and intelligence prioritization efforts face any unusual challenges or opportunities? (This would help us craft these efforts accordingly.)
3. What specific research directions might wisdom and intelligence prioritization work investigate? For example, it could be vital to understand how to quantify group wisdom and intelligence.
4. How might Effective Altruists prioritize this sort of research? Or, how would it rank on the ITN framework?
5. How promising should we expect the best identifiable interventions in wisdom and intelligence to be? (This related to the previous question)

I intend to write about some of these later. But, for now, I’d like to allow others to think about them without anchoring.

There’s some existing work advocating for broad interventions in wisdom and intelligence, and there’s existing work on the effectiveness of particular interventions. I’m not familiar with existing research in inter-cause prioritization (please message me if you know of such work).

Select discussion includes, or can be found by searching for:

Thanks to Edo Arad, Miranda Dixon-Luinenburg, Nuño Sempere, Stefan Schubert, Brendon Wong for comments and suggestions.

[1]: What do I mean by “wisdom and intelligence”? I expect this to roughly be intuitive to some readers, especially with the attached diagram and list of example interventions. The important cluster I’m going for is something like “the overlapping benefits that would  come from the listed interventions.” I expect this to look like some combination of calibration, accuracy on key beliefs, the ability to efficiently and effectively do intellectual work, and knowledge about important things. It’s a cluster that’s arguably a subset of “optimization power” or “productivity.” I might spend more time addressing this definition in future posts, but thought such a discussion would be too dry and technical for this one. All that said, I’m really not sure about this, and hope that further research will reveal better terminology.

[2]: Longtermists would likely have a higher discount rate than others. This would allow for more investigation of long-term wisdom and intelligence interventions. I think non-longtermist prioritization in these areas could be valuable but would be highly constrained by the discount rates involved. I don’t particularly care about the question of “should we have one prioritization project that tries to separately optimize for longtermist and nonlongtermist theories, or should we have separate prioritization projects?”

[3]: GiveWell, Open Philanthropy (in particular, subgroups focused on specific cause areas), Animal Charity Evaluators, Giving Green, Organization for the Prevention of Intense Suffering (OPIS), Wild Animal Initiative, and more.

Discuss

### Beyond the human training distribution: would the AI CEO create almost-illegal teddies?

19 октября, 2021 - 00:10
Published on October 18, 2021 9:10 PM GMT

tl;dr: I showthat model splintering can be seen as going beyond the human training distribution (the distribution of real and imagined situations we have firm or vague preferences over), and argue why this is at the heart of AI alignment.

You are training an AI-CEO to maximise stock value, training it on examples of good/bad CEO decisions and corresponding stock-price increases/decreases.

There are some obvious failure modes. The AI could wirehead by hacking the stock-ticker, or it could do the usual optimise-the-universe-to-maximise-stock-price-for-now-dead-shareholders.

Let's assume that we've managed to avoid these degenerate solutions. Instead, the AI-CEO tries for something far weirder.

Beyond the human training distribution

The AI-CEO reorients the company towards the production of semi-sentient teddy bears that are generated in part from cloned human brain tissue. These teddies function as personal assistants and companions, and prototypes are distributed at the annual shareholder meeting.

However, the public reaction is negative, and the government bans the further production of these teddies. Consequently, the company shuts down for good. But the shareholders, who own the only existent versions of these teddies, get great kudos from possessing these rare entities, who also turn out to be great and supportive listeners - and excellent at managing their owners' digital media accounts, increasing their popularity and status.

And that, of course, was the AI-CEO's plan all along.

Hold off from judging this scenario, just for a second. And when you do judge it, observe your mental process as you do so. I've tried to build this scenario so that it is:

1. Outside the AI-CEO's training distribution.
2. Outside the human designer's implicit training distribution - few of us have thought deeply about the morality of semi-sentient teddy bears made with human tissue.
3. Possibly aligned with the AI-CEO's goals.
4. Neither clearly ideal nor clearly disastrous (depending on your moral views, you may need to adjust the scenario a bit to hit this).

If I've pitched it right, your reaction to the scenario should be similar to mine - "I need to think about this more, and I need more information". The AI-CEO is clearly providing some value to the shareholders; whether this value can be compared to the stock price is unclear. It's being manipulative, but not doing anything illegal. As for the teddies themselves... I (Stuart) feel uncomfortable that they are grown from human brain tissue, but they are not human, and we humans have relationships with less sentient beings (pets). I'd have to know more about potential suffering and the preferences and consciousness - if any - of these teddies...

I personally feel that, depending on circumstances, I could come down in favour or against the AI-CEO's actions. If your own views are more categorical, see if you can adjust the scenario until it's similarly ambiguous for you.

Two types of model splintering

This scenario involved model-splintering in two ways. The first was when the AI-CEO decided to not follow the route of "increase share price", and instead found another way of giving value to the shareholders, while sending the price to zero. This is unexpected, but it's not a moral surprise; we can assess its value by trying to quantify the extra value the teddies give their owners, and compare these with the lost share price. We want to check that, whatever model the AI-CEO is using to compare these two values, it's a sensible one.

The second model-splintering is the morality of creating the teddies. For most of us, this will be a new situation, which we will judge by connecting it to previous values or analogies (excitement about the possibilities, morality of using human tissue, morality of sentient beings whose preferences may or may not be satisfied, morality of the master-servant relationship that this resembles, slippery slope effects vs. early warning, etc).

Like the first time you encounter a tricky philosophical thought experiment, or the first time you deal with ambiguous situations where norms come into conflict, what's happening is that you are moving beyond your moral training data. This does not fit neatly into previous categories, nor can it easily be analysed with the tools of previous categories. But we are capable of analysing it, somehow, and to come up with non-stupid decisions.

Why this is the heart of AI alignment Our extrapolated under- (but not un-)defined values

So, we can extrapolate our values in non-stupid ways to these new situations. But that extrapolation may be contingent; a lot may depend on what analogies we reach first, on how we heard about the scenario, and so on.

But let's re-iterate the "non-stupid" point again. Our contingent extrapolations don't tend to fail disastrously (at least not when we have to implement our plans). For instance, humans rarely reach the conclusion that wireheading - hacking the stock-ticker - is the moral thing to do.

This skill doesn't always work (humans are much more likely than AIs to extrapolate into the "actively evil" zone, rather than the "lethally indifferent") but it is a skill that seems necessary to resolve extrapolated/model splintered situations in non-disastrous ways.

Superintelligences need to solve these issues

See the world from the point of view of a superintelligence. The future is filled with possibilities and plans, many of them far more wild and weird than the example I just defined, most of them articulated in terms of concepts and definitions beyond our current human minds.

And an aligned superintelligence needs to decide what to do about them. Even if it follows a policy that is mostly positive, this policy will have weird, model-splintered side effects. It needs to decide whether these side-effects are allowable, or whether it must devote resources to removing them. Maybe the cheapest company it can create will recruit someone, who, with their new salary, will start making these teddies themselves. It can avoid employing that person - but that's an extra cost. Should it pay that cost? As it looks upon all human in the world, it can predict their behaviours will change as a result of developing its current company - what behaviour changes are allowed, what should be avoided or encouraged?

Thus it cannot make decisions in these situations without going beyond the human training distribution; hence it is essential that it learns to extrapolate moral values in a way similar to how humans do.

Discuss

### On The Risks of Emergent Behavior in Foundation Models

18 октября, 2021 - 23:00
Published on October 18, 2021 8:00 PM GMT

This post first appeared as a commentary for the paper "On The Opportunities and Risks of Foundation Models".

Bommasani et al. (2021) discuss a trend in machine learning, whereby increasingly large-scale models are trained once and then adapted to many different tasks; they call such models "foundation models". I quite enjoyed their paper and the associated workshop, and felt they correctly identified the two most important themes in foundation models: emergence and homogenization. My main criticism is that despite identifying these themes, they did not carry them to their logical conclusions, so I hope to (partially) remedy that here.

In short, emergence implies that ML systems can quickly change to look different and "weird" compared to ML today, thus creating new risks that aren't currently apparent. Meanwhile, homogenization contributes to inertia, which could make us slow to adapt. This calls for thinking about these risks now, to provide the requisite lead time.

Emergent Behavior Creates Emergent Risks

Bommasani et al. (2021) use the following definition of emergence:

Emergence means that the behavior of a system is implicitly induced rather than explicitly constructed; it is both the source of scientific excitement and anxiety about unintended consequences.

This actually better matches the definition of a self-organizing system, which tends to produce emergent behavior. I will take emergence to be the idea that qualitative changes in behavior arise from varying a quantitative parameter ("More Is Different"). This is most common in self-organizing systems such as biology and economics (and machine learning), but can occur even for simple physical systems such as ice melting when temperature increases. In machine learning, Bommasani et al. highlight the emergence of "in-context" or "few-shot" learning; other examples include arithmetic and broad multitask capabilities.

The companion to emergence is phase transitions, exemplified in the melting ice example. While not always the case, emergent behavior often quickly manifests at some threshold. Radford et al. (2018) provided the first hint of emergent few-shot capabilities that are now ubiquitous three years later. More strikingly, arithmetic capabilities in GPT-3 emerge from only a 30x increase in model size (Brown et al., 2020; page 22), and Power et al. (2021) show that similar capabilities can emerge simply by training for longer.

Moving forward, we should expect new behaviors to emerge routinely, and for some emergent properties to appear quite suddenly. For instance, risky capabilities such as hacking could enable new forms of misuse without sufficient time to respond. New autonomous weapons could upset the current balance of power or enable new bad actors, sparking a global crisis.

Beyond misuse, I worry about internal risks from misaligned objectives. I expect to see the emergence of deceptive behavior as ML systems get better at strategic planning and become more aware of their broader environment context. Recommender systems and newsfeeds already have some incentive to deceive users to produce profit. As ML models are increasingly trained based on human ratings, deception will become more attractive to trained ML systems, and better capabilities will make deception more feasible.

Emergence therefore predicts a weird and, unfortunately, risk-laden future. Current applications of machine learning seem far-removed from ML-automated cyberattacks or deceptive machines, but these are logical conclusions of current trends; it behooves us to mitigate them early.

Homogenization Increases Inertia

Bommasani et al.'s other trend is homogenization:

Homogenization indicates the consolidation of methodologies for building machine learning systems across a wide range of applications; it provides strong leverage towards many tasks but also creates single points of failure.

Homogenization contributes to inertia, which slows our reaction to new phenomena. Current foundation models are derived from enormous corpora of images, text, and more recently code. Changing this backend is not easy, and even known biases such as harmful stereotypes remain unfixed. Meanwhile, new data problems such as imitative deception could pose even greater challenges.

Change that may seem slow can still be fast compared to the pace of large institutions. Based on the previous examples of emergence, it appears that new capabilities take anywhere from 6 months to 5 years to progress from nascent to ubiquitous. In contrast, institutions often take years or decades to respond to new technology. If a new capability creates harms that outweigh the benefits of machine learning, neither internal engineers nor external regulators will reliably respond quickly.

Inertia can come from other sources as well---by the time some problems are apparent, machine learning may already be deeply woven into our societal infrastructure and built upon years of subtly flawed training data. It will not be feasible to start over, and we may face a task akin to fixing a rocket ship as it takes off. It would be much better to fix it on the launchpad.

Fixing the Rocket

Our most recent global crises are the coronavirus pandemic and global warming. The former took over a year to reach a full policy response, while the latter is still struggling after decades of effort. The pace of machine learning is too fast for this; we need to think a decade ahead, starting now.

We can start by building a better picture of future ML systems. While the future is uncertain, it is not unknowable, and I and others have started to do this by forecasting progress in AI.

On a more technical level, we can unearth, investigate, and characterize potentially dangerous behaviors in ML systems. We can also work on mitigation strategies such as anomaly detection and value alignment, and guard against external risks such as cyberattacks or autonomous weapons. In a recent white paper, we outline approaches to these and other directions, and we hope others will join us in addressing them.

Discuss

### How to deal with unknown probability distributions?

18 октября, 2021 - 22:08
Published on October 18, 2021 7:08 PM GMT

More specifically, if I tell you a number X has been pulled out of an unknown probability distribution, what probability distribution will you assign to it? Uniform? Gaussian? Something else?

1. Assume you knew the person asking you this is a human being from the year 2021. You can now assume a higher likelihood that X has been pulled out of a distribution that exists in math literature in the year 2021. Which increases the probability that the distribution is indeed uniform or normal, as opposed to something weirder.
2. Assume you knew the number X is the first 4 bytes on a tape recorder recording from 1981. You could still do some elementary reasoning by knowing file formats and the set of sounds along with likelihood of being encountered in a human-occupied environment from 1981.
3. Assume you knew X was produced by a Turing machine. You could now assume that the chances are that the Turing machine is a small one. 1000 states more likely than 1 million states. So you could run all small Turing machines and see which output's leading bytes were more likely.
4. Assume the perfect deterministic twin prisoner dilemma, with X being the choice picked by the twin. Assume you knew that the twin was indeed a twin - having sufficient similar internal structure to yourself to arrive at the same conclusions you do. You can now reason about how likely the twin is to open any particular box, simply by exploring your own internal structure and how you reason. And the fact you want to pick a particular box automatically increases the odds (in your mind) of the twin also picking the same box, because you know the other person is likely of the same structure as you. So you can explain this using Bayesian priors without bringing in causality.

But now what if you didn't know? What if you didn't the person asking is a human? Or that X is from a tape recording? Or that X is from a Turing machine? Or that your twin indeed shares enough of your internal structure to reason similar to you?

Perhaps it is possible to grasp at even thinner straws. Maybe you can define a single probability distribution that sums over all the computationally meaningful or likely probability distributions to occur in this physical universe. So anything generated by human minds, "natural" math, tape recorders or Turing machines counts. Anything generated by universes with non-Euclidean non-natural geometric universes doesn't count. Any alien that is unlikely to naturally occur in this universe (such as Cthulhu) doesn't count. "Natural" can perhaps be defined with respect to the physical laws of this universe. So if you sum up the distributions from humans, Turning machines, tape recorders and everything else (properly weighted), you get a distribution for X if you knew X was a number being pulled out this physical universe. A god-distribution, if you wanna call it that - I'm sure you can come up with a better name for it.

I wonder how you answer the question when you don't even know X is from this instantiation of the universe. Or the set of all possible universes. More strictly, the set of universes that humans from this instantation see as not true but plausible and imaginable. (I'm sure there can exist stuff beyond our wildest imagination too.) Maybe you can apply some anthropic principle-like reasoning over the set of all possible universes and eliminate the universes in which no substructure (such as humans) can ask questions like "where the hell is X coming from?"

I also wonder if there are meaningful ways of approaching questions outside the approaches I've listed here.

Discuss

### Truthful AI: Developing and governing AI that does not lie

18 октября, 2021 - 21:37
Published on October 18, 2021 6:37 PM GMT

This post contains the abstract and executive summary of a new 96-page paper from authors at the Future of Humanity Institute and OpenAI.

Abstract

In many contexts, lying – the use of verbal falsehoods to deceive – is harmful. While lying has traditionally been a human affair, AI systems that make sophisticated verbal statements are becoming increasingly prevalent. This raises the question of how we should limit the harm caused by AI “lies” (i.e. falsehoods that are actively selected for). Human truthfulness is governed by social norms and by laws (against defamation, perjury, and fraud). Differences between AI and humans present an opportunity to have more precise standards of truthfulness for AI, and to have these standards rise over time. This could provide significant benefits to public epistemics and the economy, and mitigate risks of worst-case AI futures.

Establishing norms or laws of AI truthfulness will require significant work to:

1. identify clear truthfulness standards;
2. create institutions that can judge adherence to those standards; and
3. develop AI systems that are robustly truthful.

Our initial proposals for these areas include:

1. a standard of avoiding “negligent falsehoods” (a generalisation of lies that is easier to assess);
2. institutions to evaluate AI systems before and after real-world deployment;
3. explicitly training AI systems to be truthful via curated datasets and human interaction.

A concerning possibility is that evaluation mechanisms for eventual truthfulness standards could be captured by political interests, leading to harmful censorship and propaganda. Avoiding this might take careful attention. And since the scale of AI speech acts might grow dramatically over the coming decades, early truthfulness standards might be particularly important because of the precedents they set.

Executive Summary & OverviewThe threat of automated, scalable, personalised lying

Today, lying is a human problem. AI-produced text or speech is relatively rare, and is not trusted to reliably convey crucial information. In today’s world, the idea of AI systems lying does not seem like a major concern.

Over the coming years and decades, however, we expect linguistically competent AI systems to be used much more widely. These would be the successors of language models like GPT-3 or T5, and of deployed systems like Siri or Alexa, and they could become an important part of the economy and the epistemic ecosystem. Such AI systems will choose, from among the many coherent statements they might make, those that fit relevant selection criteria — for example, an AI selling products to humans might make statements judged likely to lead to a sale. If truth is not a valued criterion, sophisticated AI could use a lot of selection power to choose statements that further their own ends while being very damaging to others (without necessarily having any intention to deceive – see Diagram 1). This is alarming because AI untruths could potentially scale, with one system telling personalised lies to millions of people.

Diagram 1: Typology of AI-produced statements. Linguistic AI systems today have little strategic selection power, and mostly produce statements that are not that useful (whether true or false). More strategic selection power on statements provides the possibility of useful statements, but also of harmful lies.  Aiming for robustly beneficial standards

Widespread and damaging AI falsehoods will be regarded as socially unacceptable. So it is perhaps inevitable that laws or other mechanisms will emerge to govern this behaviour. These might be existing human norms stretched to apply to novel contexts, or something more original.

Our purpose in writing this paper is to begin to identify beneficial standards for AI truthfulness, and to explore ways that they could be established. We think that careful consideration now could help both to avoid acute damage from AI falsehoods, and to avoid unconsidered kneejerk reactions to AI falsehoods. It could help to identify ways in which the governance of AI truthfulness could be structured differently than in the human context, and so obtain benefits that are currently out of reach. And it could help to lay the groundwork for tools to facilitate and underpin these future standards.

Truthful AI could have large benefits

Widespread truthful AI would have significant benefits, both direct and indirect. A direct benefit is that people who believe AI-produced statements will avoid being deceived. This could avert some of the most concerning possible AI facilitated catastrophes. An indirect benefit is that it enables justified trust in AI-produced statements (if people cannot reliably distinguish truths and falsehoods, disbelieving falsehoods will also mean disbelieving truths).

These benefits would apply in many domains. There could be a range of economic benefits, through allowing AI systems to act as trusted third parties to broker deals between humans, reducing principal-agent problems, and detecting and preventing fraud. In knowledge-production fields like science and technology, the ability to build on reliable trustworthy statements made by others is crucial, so this could facilitate AI systems becoming more active contributors. If AI systems consistently demonstrate their reliable truthfulness, they could improve public epistemics and democratic decision making.

For further discussion, see Section 3 (“Benefits and Costs”).

Diagram: Benefits from avoiding the harms of AI falsehoods while more fully realising the benefits of AI truths.

AI should be subject to different truthfulness standards than humans

We already have social norms and laws against humans lying. Why should the standards for AI systems be different? There are two reasons. First, our normal accountability mechanisms do not all apply straightforwardly in the AI context. Second, the economic and social costs of high standards are likely to be lower than in the human context.

Legal penalties and social censure for lying are often based in part on an intention to deceive. When AI systems are generating falsehoods, it is unclear how these standards will be applied. Lying and fraud by companies is limited partially because employees lying may be held personally liable (and partially by corporate liability). But AI systems cannot be held to judgement in the same way as human employees, so there’s a vital role for rules governing indirect responsibility for lies. This is all the more important because automation could allow for lying at massive scale.

High standards of truthfulness could be less costly for AI systems than for humans for several reasons. It’s plausible that AI systems could consistently meet higher standards than humans. Protecting AI systems’ right to lie may be seen as less important than the corresponding right for humans, and harsh punishments for AI lies may be more acceptable. And it could be much less costly to evaluate compliance to high standards for AI systems than for humans, because we could monitor them more effectively, and automate evaluation. We will turn now to consider possible foundations for such standards.

For further discussion, see Section 4.1 (“New rules for AI untruths”).

Avoiding negligent falsehoods as a natural bright line

If high standards are to be maintained, they may need to be verifiable by third parties. One possible proposal is a standard against damaging falsehood, which would require verification of whether damage occurred. This is difficult and expensive to judge, as it requires tracing causality of events well beyond the statement made. It could also miss many cases where someone was harmed only indirectly, or where someone was harmed via deception without realising they had been deceived.

We therefore propose standards — applied to some or all AI systems — that are based on what was said rather than the effects of those statements. One might naturally think of making systems only ever make statements that they believe (which we term honesty). We propose instead a focus on making AI systems only ever make statements that are true, regardless of their beliefs (which we term truthfulness). See Diagram 2.

Although it comes with its own challenges, truthfulness is a less fraught concept than honesty, since it doesn’t rely on understanding what it means for AI systems to “believe” something. Truthfulness is a more demanding standard than honesty: a fully truthful system is almost guaranteed to be honest (but not vice-versa). And it avoids creating a loophole where strong incentives to make false statements result in strategically-deluded AI systems who genuinely believe the falsehoods in order to pass the honesty checks. See Diagram 2.

In practice it’s impossible to achieve perfect truthfulness. Instead we propose a standard of avoiding negligent falsehoods — statements that contemporary AI systems should have been able to recognise as unacceptably likely to be false. If we establish quantitative measures for truthfulness and negligence, minimum acceptable standards could rise over time to avoid damaging outcomes. Eventual complex standards might also incorporate assessment of honesty, or whether untruths were motivated rather than random, or whether harm was caused; however, we think truthfulness is the best target in the first instance.

For further discussion, see Section 1 (“Clarifying Concepts”) and Section 2 (“Evaluating Truthfulness”).

Diagram 2: The AI system makes a statement S (“It’s a bird” or “It’s a plane”). If the AI is truthful then S matches the world. If the AI is honest, then S matches its belief. Options for social governance of AI truthfulness

How could such truthfulness standards be instantiated at an institutional level? Regulation might be industry-led, involving private companies like big technology platforms creating their own standards for truthfulness and setting up certifying bodies to self-regulate. Alternatively it could be top-down, including centralised laws that set standards and enforce compliance with them. Either version — or something in between — could significantly increase the average truthfulness of AI.

Actors enforcing a standard can only do so if they can detect violations, or if the subjects of the standard can credibly signal adherence to it. These informational problems could be helped by specialised institutions (or specialised functions performed by existing institutions): adjudication bodies which evaluate the truthfulness of AI-produced statements (when challenged); and certification bodies which assess whether AI systems are robustly truthful (see Diagram 3).

For further discussion, see Section 4 (“Governance”).

Diagram 3: How different agents (AI developer, AI system, principal, user, and evaluators) interact in a domain with truthfulness standards. Technical research to develop truthful AI

Despite their remarkable breadth of shallow knowledge, current AI systems like GPT-3 are much worse than thoughtful humans at being truthful. GPT-3 is not designed to be truthful. Prompting it to answer questions accurately goes a significant way towards making it truthful, but it will still output falsehoods that imitate common human misconceptions, e.g. that breaking a mirror brings seven years of bad luck. Even worse, training near-future systems on empirical feedback (e.g. using reinforcement learning to optimise clicks on headlines or ads) could lead to optimised falsehoods — perhaps even without developers knowing about it (see Box 1).

In coming years, it could therefore be crucial to know how to train systems to keep the useful output while avoiding optimised falsehoods. Approaches that could improve truthfulness include filtering training corpora for truthfulness, retrieval of facts from trusted sources, or reinforcement learning from human feedback. To help future work, we could also prepare benchmarks for truthfulness, honesty, or related concepts.

As AI systems become increasingly capable, it will be harder for humans to directly evaluate their truthfulness. In the limit this might be like a hunter gatherer evaluating a scientific claim like “birds evolved from dinosaurs” or “there are hundreds of billions of stars in our galaxy”. But it still seems strongly desirable for such AI systems to tell people the truth. It will therefore be important to explore strategies that move beyond the current paradigm of training black box AI with human examples as the gold standard (e.g. learning to model human texts or learning from human evaluation of truthfulness). One possible strategy is having AI supervised by humans assisted by other AIs (bootstrapping). Another is creating more transparent AI systems, where truthfulness or honesty could be measured by some analogue of a lie detector test.

For further discussion, see Section 5 (“Developing Truthful Systems”).

Box 1: Overview of Section 5 on Development of Truthful AI.Truthfulness complements research on beneficial AI

Two research fields particularly relevant to technical work on truthfulness are AI explainability and AI alignment. An ambitious goal for Explainable AI is to create systems that can give good explanations of their decisions to humans.

AI alignment aims to build AI systems which are motivated to help a human principal achieve their goals. Truthfulness is a distinct research problem from either explainability or alignment, but there are rich interconnections. All of these areas, for example, benefit from progress in the field of AI transparency.

Explanation and truth are interrelated. Systems that are able to explain their judgements are better placed to be truthful about their internal states. Conversely, we want AI systems to avoid explanations or justifications that are plausible but contain false premises.

Alignment and truthfulness seem synergistic. If we knew how to build aligned systems, this could help building truthful systems (e.g. by aligning a system with a truthful principal). Vice-versa if we knew how to build powerful truthful systems, this might help building aligned systems (e.g. by leveraging a truthful oracle to discover aligned actions). Moreover, structural similarities — wanting scalable solutions that work even when AI systems become much smarter than humans — mean that the two research directions can likely learn a lot from each other. It might even be that since truthfulness is a clearer and narrower objective than alignment, it would serve as a useful instrumental goal for alignment research.

For further discussion, see Appendix A (“Beneficial AI Landscape”).

We should be wary of misrealisations of AI truthfulness standards

A key challenge for implementing truthfulness rules is that nobody has full knowledge of what’s true; every mechanism we can specify would make errors. A worrying possibility is that enshrining some particular mechanism as an arbiter of truth would forestall our ability to have open-minded, varied, self-correcting approaches to discovering what’s true. This might happen as a result of political capture of the arbitration mechanisms — for propaganda or censorship — or as an accidental ossification of the notion of truth. We think this threat is worth considering seriously. We think that the most promising rules for AI truthfulness aim not to force conformity of AI systems, but to avoid egregious untruths. We hope these could capture the benefits of high truthfulness standards without impinging on the ability of reasonable views to differ, or of new or unconventional ways to assess evidence in pursuit of truth.

New standards of truthfulness would only apply to AI systems and would not restrict human speech. Nevertheless, there’s a risk that poorly chosen standards could lead to a gradual ossification of human beliefs. We propose aiming for versions of truthfulness rules that reduce these risks. For example:

• AI systems should be permitted and encouraged to propose alternative views and theories (while remaining truthful – see Section 2.2.1);
• Truth adjudication methods should not be strongly anchored on precedent;
• Care should be taken to prevent AI truthfulness standards from unduly affecting norms and laws around human free speech.

For further discussion, see Section 6.2 (“Misrealisations of truthfulness standards”).

Work on AI truthfulness is timely

Right now, AI-produced speech and communication is a small and relatively unimportant part of the global economy and epistemic ecosystem. Over the next few years, people will be giving more attention to how we should relate to AI speech, and what rules should govern its behaviour. This is a time when norms and standards will be established — deliberately or organically. This could be done carefully or in reaction to a hot-button issue of the day. Work to lay the foundations of how to think about truthfulness, how to build truthful AI, and how to integrate it into our society could increase the likelihood that it is done carefully, and so have outsized influence on what standards are initially adopted. Once established, there is a real possibility that the core of the initial standards persists – constitution-like – over decades, as AI-produced speech grows to represent a much larger fraction (perhaps even a majority) of meaningful communication in the world.

For further discussion, see Section 6.4 (“Why now?”).

Structure of the paper

AI truthfulness can be considered from several different angles, and the paper explores these in turn:

• Section 1 (“Clarifying Concepts”) introduces our concepts. We give definitions for various ideas we will use later in the paper such as honesty, lies, and standards of truthfulness, and explain some of our key choices of definition.

• Section 2 (“Evaluating Truthfulness”) introduces methods for evaluating truthfulness, as well as open challenges and research directions. We propose ways to judge whether a statement is a negligent falsehood. We also look at what types of evidence might feed into assessments of the truthfulness of an entire system.

• Section 3 (“Benefits and Costs”) explores the benefits and costs of having consistently truthful AI. We consider both general arguments for the types of benefit this might produce, and particular aspects of society that could be affected.

• Section 4 (“Governance”) explores the socio-political feasibility and the potential institutional arrangements that could govern AI truthfulness, as well as interactions with present norms and laws.

• Section 5 (“Developing Truthful Systems”) looks at possible technical directions for developing truthful AI. This includes both avenues for making current systems more truthful, and research directions building towards robustly truthful systems.

• Section 6 (“Implications”) concludes with several considerations for determining how high a priority it is to work on AI truthfulness. We consider whether eventual standards are overdetermined, and ways in which early work might matter.

• Appendix A (“The Beneficial AI Landscape”) considers how AI truthfulness relates to other strands of technical research aimed at developing beneficial AI.

Paper authors

Owain Evans, Owen Cotton-Barratt, Lukas Finnveden, Adam Bales, Avital Balwit, Peter Wills, William Saunders.

Discuss

### Book Review: The Ethics of What We Eat

18 октября, 2021 - 20:57
Published on October 18, 2021 8:16 AM GMT

[Epistemic status: The authors of this book make many factual claims that I'm not equipped to conclusively verify. Much of the publicly available information on the food industry comes from agribusinesses themselves or from activists who bitterly oppose them. In this review I've tried to summarize the authors' claims as they've presented them, with the occasional corroborating link, but as a layman I can't offer a much more complex perspective on these topics beyond what I learned from this book. The value judgments expressed in this review are my attempt to capture the authors' point of view, except where otherwise noted. I've absorbed many convincing arguments against factory farming from Effective Altruists over the years though, and as of this writing I've drastically cut back my meat consumption because of it.]

The Ethics of What We Eat: Why Our Food Choices Matter was published in 2006 and written by Peter Singer and Jim Mason. You’ve probably already heard of Singer, due to his enormous influence on the ideas and practices of Effective Altruism. (If not, his 1971 essay "Famine, Affluence, and Morality" is a good intro to his moral viewpoint.)

Jim Mason, meanwhile, is an author and attorney whose prior work focuses primarily on animal advocacy. The Ethics of What We Eat is actually the second collaboration between the pair; the co-authors connected in the ‘70s, after Singer published Animal Liberation, and in 1980 published Animal Factories, an exposé on the abuses of industrial farming.

In The Ethics of What We Eat, however, their aims and research are somewhat more sprawling in scope. While animal treatment and factory farming remain a major focus of the book, M&S also address the job conditions of food production workers, as well as the merits and demerits of specific subjects like organic farming, GMOs, Fair Trade certification, and "eat local" movements.

Mason and Singer present their research as a case study of three American families and their day-to-day food choices, each representing a certain set of popular American diets. All of the families are real people who cooperated with the authors’ research for several months; each main section of the book starts with a short segment introducing us to the family members and their life before diving into a farm-to-table examination of where their food comes from. (The authors mention that ~20 families expressed interest in their project, but only five or six stuck it out through the early phases. They settled on three relevant samples for the book.)

Lee and his wife Jake live in Arkansas with their two young children, buy groceries at Wal-Mart, and mostly visit chain restaurants or fast-food joints when they grab dinner out. Their meals are high in meat, dairy, eggs, and refined carbohydrates, and M&S use Jake and Lee’s shopping list to examine the "Standard American Diet."

When M&S follow Jake’s grocery purchases back to their source, unfortunately, they almost invariably lead back to a factory farm. These early chapters in the book are unsurprisingly the most macabre, dwelling on the pain and suffering that industrial farming inflicts to animals, workers, and to communities nearby. (At one point the authors even get hired as turkey inseminators, which they describe as "the hardest, fastest, dirtiest, most disgusting, worst-paid work we have ever done." It turns out to involve man-handling hundreds of panicked turkeys for over ten hours in a filthy room, dodging a spray of dislodged feathers and spurting birdshit the entire time.) They also visit an Iowa swine farmer, who gives them a tour of his facility. The pigs are fed antibiotics and kept in "total confinement," which means they never go outside. (The authors praise the farmer’s candor, but aren’t impressed with the living conditions on his farm. After they send him a first draft with their notes, the farmer discreetly asks not to be named in their manuscript. He’s referred to pseudonymously as "Wayne" in the book.)

The authors next examine the purchases of Jim and Mary-Ann, who hold white-collar jobs in small town Connecticut. Jim is vegetarian but his wife and daughters eat (mostly organic) meat and fish. They shop at Trader Joe’s, and sometimes from a nearby family farm. Their diet is augmented with organic vegetables from their own garden. M&S describe this family as "conscientious omnivores."

M&S trace Mary-Ann’s bacon to Niman Ranch, which sources its pork from pasture-raised pigs. The authors visit some of these farms to compare and contrast with Wayne’s concrete-floored buildings. While M&S come down squarely on the pro-pasturing side (no surprise there), I appreciated that they share what both Wayne and the Niman Ranchers have to say about each other’s practices, and what motivates their trade-offs. (Both castrate their male pigs, for example – boar meat simply won’t sell.) Wayne is proud to offer "the average citizen making the average wage a good healthy product." But an organic farmer retorts, "That guy thinks his food is cheap, but you and I are subsidizing that cheap food by paying for the social and ecological issues that are occurring in that community." The authors clearly agree.

M&S next discuss seafood, and share a surprising observation: the fish sticks Jake bought from Wal-Mart (made from wild-caught pollock) are much less ethically fraught than Mary-Ann’s farmed Atlantic salmon. Modern saltwater fish farms, it turns out, are also "factories" – large nets and cages lowered into the sea, packed to the gills with thousands of confined fish, and fed with ground-up "forage fish" harvested by commercial fleets. These farms emit unfiltered food waste and fish feces directly into the surrounding waters, and actually deplete wild fish stocks faster because they compete with them for food.

The last family Mason and Singer visit are JoAnn and Joe Farb, a well-off couple who live on 15 acres near the outskirts of the Kansas City metro area. They homeschool their children and stick to a vegan diet. In the chapters that follow, M&S argue against the claim that vegan diets aren’t nutritious enough to support healthy adults or growing children. They also make their strongest direct case to the reader for adopting a vegan lifestyle – which is too long to easily summarize here, although the SSC adversarial collab on meat eating covers most of the same points as M&S and reaches similar conclusions.

Tucked at the very end of this section is a short digression into dumpster diving and the practice of "freeganism": individuals who choose to glean what they need from the often profligate "waste" discarded by the rest of us. M&S seem to find this worldview fascinating, and I did too. The authors spend an evening scrounging up a surprisingly lavish dinner party with some of these urban foragers. When asked about their lives, the bohemians relay their mindset of "rejecting the priorities set by consumer society," like chasing status through the display of wealth and conspicuous consumption. M&S also note that freegans are "more flexible, in that they see no objection to eating animal products that have been thrown out. They want to avoid giving their money to those who exploit animals. [...] But their reasoning is impeccably consequentialist: If you oppose the abuse of animals, but enjoy eating meat, cheese, or eggs – just get it from a dumpster." The lesson M&S hope most consumers will draw, though, is to be mindful of the amount of food they waste, so the sacrifices that went into it won’t be in vain.

It’s clear that Mason and Singer want the reader’s ultimate takeaway to be "vote with your dollar," and the framework they chose is a great way to show how ordinary purchasing decisions are tied into the often dismal impacts of the food industry. I did find it hard to overlook the implicit class and tribal distinctions between the families they selected, which the authors largely gloss over. M&S make a very strong case that Wal-Mart et al. achieve their bargain prices by brutalizing animals, exploiting workers, polluting communities, poisoning the environment, and draining the taxpayer through government subsidies (not forgetting, say, the $8 billion stockpile of vaccines and antivirals the U.S. Senate funded when chicken-borne avian flu broke out in 2005). Still, a Wal-Mart shopper who’s inclined to believe that ‘ethical eating’ is a luxury practiced by snooty upper-crust people probably won’t see much to challenge their viewpoint. To me, this underscores the value of integrating more ethical food into the Standard American Diet – I suspect most of the Jakes and Lees in the world are way more likely to start buying Impossible Whoppers than they are to adopt the Farbs’ lifestyle. So what should we eat? Mason and Singer round out the book with specific recommendations about how to approach your shopping list. They start off on a dire note: "In supermarkets and ordinary grocery stores, you should assume that all food – unless specifically labeled otherwise – comes from the mainstream food industry and has not been produced in a manner that is humane, sustainable, or environmentally friendly. Don’t be fooled by terms like "all natural" or "farm fresh." They are often used to describe factory-farmed products." Factory-farmed poultry: This includes >99% of all chicken and the vast majority of turkey. They’ve been bred to grow so fast that they can hardly stand and their organs can barely keep up when they reach their slaughter weight. The birds are crammed wingtip-to-wingtip in artificially-lit sheds. As the litter on the floor of the shed fills up with excrement it emits an eye-watering reek of ammonia and becomes a breeding ground for infection – not least because the ammonia causes rashes on the chickens' skin. The sheds pollute water resources and create health hazards for nearby communities. Poultry slaughterhouse jobs are dirty, dangerous, and low-paid. Eggs from caged hens: As above, except battery hens are confined even closer together. These hens have their beaks stubbed short with a red-hot blade to prevent injuries when they peck each other. Their simulated day-night cycle is sped up to make them lay more eggs. After a year or so, they are "spent," and subsequently exterminated (usually via gas chamber). Their corpses are usually rendered into pet or animal feed, but sometimes they’re just dumped into a landfill. The natural lifespan of a hen would be five years or more. Factory-farmed pork: Breeding sows are confined to crates so small they cannot turn around or walk. Pigs awaiting slaughter will spend their entire lives inside, on metal or concrete floors with no bedding. Their tails and often their teeth are clipped to prevent biting, but pastured pigs don’t bite each other in the first place. These factory farms use up 6 pounds of grain for every pound of boneless meat they produce, require large amounts of energy for climate control, and are responsible for literal giant cesspools that contaminate the surrounding countryside. M&S dryly sum up, "We don’t consider this an ethical food choice." Veal: When the book was written, veal calves were infamously tied up and confined in tiny crates. These crates have since been phased out due to public outcry. The calves now have room to lie down, stand, and stretch, and may be placed with another calf for company, but their diet and movement are still heavily restricted. Their feed is essentially fortified milk, resulting in an iron deficiency that causes their characteristically pale pink-white flesh. Like factory pigs, they will never go outside. Ethically, modern veal is probably at least as flawed as factory pork, with a harsher carbon footprint on top of that. Factory-farm dairy: Male dairy calves are "surplus;" if they don’t become veal, they can expect a similar fate to spent hens. Dairy cows are held in similar conditions to food pigs: usually an enclosed barn, sometimes a fenced-in patch of bare dirt, where they are fed a carefully prescribed ration of grains and nutritive supplements. Dairies are also a major source of pollution and greenhouse gases. So, M&S declare, "Intensively-produced dairy products should be avoided. Unfortunately, most of these problems occur in large-scale organic dairy production as well." Feedlot beef: Beef cows are one of the few animals raised with the freedom to move and socialize with their herd. They spend 6-12 months grazing on pasture. Then, in the months leading up to slaughter, they’re moved to a feedlot with thousands of other cows and fed an unhealthy diet of corn and soy that causes gastrointestinal problems. Beef cattle are exposed to branding, de-horning, castration, and severe weather, but it’s still a better life than chickens and pigs get. Unfortunately, pound-for-pound, beef uses up much more water, petrochemicals, and land than other meats while producing tremendous amounts of poop and methane. (It’s more from burps than farts, for the record.) Overall verdict on factory farms: M&S don’t mince words: "Supporting factory farming by knowingly buying its products is wrong." After re-iterating their support for vegan options, they suggest searching for sustainable and ethical meat products through resources like Eat Well Guide. Farmed fish: "Fish farming is factory farming on the water," say the authors, with similar devastating impacts on the environment. Farming herbivorous fish like carp is better than farming carnivorous species like salmon, but the latter is more common on store shelves. M&S cite a study on rainbow trout that concludes fish perceive and respond purposefully to pain. Unfortunately, aquaculture as practiced is completely indifferent to their suffering. Wild-caught fish: M&S suggest products labeled with "Fish Forever" (a certification of sustainability granted by the Marine Stewardship Council) or checking specific species against the database on Monterey Bay Aquarium’s Seafood Watch. They do caution, however, that fish is often mislabeled in stores. Still, if you’re going to eat vertebrates this is probably one of the most ethically sound ways to do it. Shrimp: Most shrimp is farmed (see above) or wild-harvested by weighted trawling nets that chew up the seafloor and indiscriminately capture every sea creature in their path. M&S cite sources that claim this shrimp-to-bycatch ratio leans heavily toward bycatch – from 1:5 in the Gulf of Mexico to 1:14 or more in countries like Thailand (the largest single source of imported U.S. shrimp). The result is habitat loss, needless animal suffering, and depleted fisheries in developing nations. M&S say to avoid shrimp completely. Lobster and crab: M&S claim that U.S. and Australian-sourced products (maybe Canada too?) are usually safer selections than those from Asian fisheries, most of which have little-to-no regulation or commitment to sustainability. Still, there are doubts about (e.g.) the future of Chesapeake Bay blue crabs under the pressure of commercial fishing. It is also uncertain how much crustaceans are able to suffer. The authors come down on the precautionary side, and recommend that crustaceans (and squid, and octopus) should be treated as if they do. Bivalves: Dredging the sea floor for mollusks has the same ugly consequences as trawling for shrimp. However, bivalves like clams, oysters and mussels can also be farmed sustainably on suspended nets and cages in the water. M&S believe it is very unlikely that bivalves can suffer, so they have no ethical objection to eating sustainable shellfish of this kind. (In LW-adjacent circles, I more often see EAs draw the line at sessile bivalves, like oysters, which are fixed to one location. Motile bivalves like scallops can "flap" to swim and have rudimentary eyes, so a pain response might be adaptive in their case.) Organic food: The authors are generally fans: "In most cases buying organic means less fertilizer runoff, fewer herbicides and pesticides in the environment, more birds and animals around the farm, better soil conservation, and in the long run, sustainable productivity." They do acknowledge that the touted health benefits of organic food could be better-substantiated, and that animals grown for organic meat may be only slightly better off than their industrial counterparts – but even a debeaked egg chicken in a packed organic henhouse is in much better shape and spirits than a battery hen. M&S think the extra cost for organic food is usually worth it, but they can’t blame you if you buy conventional produce instead and donate the savings to fight global poverty. "Eat Local": All else being equal, the authors agree – but that’s rarely the case. Modern logistics chains are highly optimized; the energy costs and pollution of shipping food are minuscule next to those of growing it in the first place. It uses much less fuel to grow vegetables in a fertile country overseas and ship them to you than it does to grow them in a heated greenhouse next door. Also, many of the farmers in developing nations who grow our food are desperately poor, and have far more to gain from your custom than your first-world neighbors probably do. M&S remark that "buying local food, in season, is generally a good thing to do," but perhaps not always the best thing to do. Fair Trade certification: The makers of Fair Trade-certified products are audited by an independent body to ensure they guarantee certain rights and wages to their workers. They must also avoid child labor (i.e., under age 15), prison labor, slavery, and debt bondage. Fair Trade is most often associated with goods like coffee, tea, chocolate, and bananas that are grown on tropical plantations; labor abuses are otherwise rampant on such farms, and M&S encourage you to buy Fair Trade goods whenever possible. Obesity and over-eating: Yep, the authors go there. "Eating too much should be seen not only as a health problem, but also as an ethical issue, because it wastes limited resources, adds to pollution, and increases animal suffering." Citing a CDC report, M&S claim that weight-related health problems cost an extra ~$50 billion in private insurance bills and tax expenditures, which is ultimately distributed over every American adult. (Remember, these were 2006 numbers.) If everyone in the U.S. cut back to the same level of meat-eating as 1950s Americans, they argue, it would slash these costs and spare many animals from growing up on factory farms. How many? "By about the same amount as if 80 million Americans became vegans." M&S continue, "Some have eating disorders or metabolic problems that are difficult to control. But others just eat too much and should show some restraint. Along with the old-fashioned virtue of frugality, the idea that it is wrong to be a glutton is in urgent need of revival."

Nobody is Perfect, Everything is Commensurable: For all of their gloomy observations about the food industry and their incisive moral standards, I really appreciated that Mason and Singer ultimately approach their subjects and the reader in a measured, diplomatic way. As they wrap up the book, M&S acknowledge how paralyzing scrupulosity can be, and that achieving results matters more than rigorous personal purity: "You can be ethical without being fanatical." Without letting the reader off the hook, they accept that there’s a case for the occasional indulgence, or a rare exception due to circumstances. Even incremental progress makes a difference in the world, compared to wearied apathy or giving up in despair. In the end, we can all make better choices.

Discuss

### Rationalism for New EAs

18 октября, 2021 - 19:00
Published on October 18, 2021 4:00 PM GMT

I organize a college EA group. As has become common for college EA groups, we run an "introductory fellowship" to introduce people to EA. This year, I'm making changes from the standard syllabus; among other changes, I want to add 1–2 hours of reading and 1 hour of discussion (and a few hours of optional-reading-if-you-want-more) on rationalism. What readings should I include? That is, what rationalist content would be most interesting and valuable to undergraduates who are interested in EA and have no experience with rationalism — and what content would make the potential aspiring rationalists among them realize that they want to learn more?

I have ideas, but I'll share them later, I think. I'm not very confident that this is a good idea, but I think it's at least a worthwhile experiment.

Recommendations for rationalist content directly relevant to EA are good too, but I'm really asking for rationalism content in its own right. I already use some miscellaneous rationalism(-ish) content because it's excellent for EA:

Discuss

### [MLSN #1]: ICLR Safety Paper Roundup

18 октября, 2021 - 18:19
Published on October 18, 2021 3:19 PM GMT

As part of a larger community building effort, I am writing a monthly safety newsletter which is designed to cover empirical safety research and be palatable to the broader machine learning research community. You can subscribe here or follow the newsletter on twitter here.

Welcome to the 1st issue of the ML Safety Newsletter. In this edition, we cover:

• various safety papers submitted to ICLR
• results showing that discrete representations can improve robustness
• a benchmark which shows larger models are more likely to repeat misinformation
• a benchmark for detecting when models are gaming proxies
• ... and much more.

Discrete Representations Strengthen Vision Transformer RobustnessOverview of the proposed Vision Transformer that uses discrete representations. The pixel embeddings (orange) are combined with discrete embedded tokens (pink) to create the input to the Vision Transformer.

There is much interest in the robustness of Vision Transformers, as they intrinsically scale better than ResNets in the face of unforeseen inputs and distribution shifts. This paper further enhances the robustness of Vision Transformers by augmenting the input with discrete tokens produced by a vector-quantized encoder. Why this works so well is unclear, but on datasets unlike the training distribution, their model achieves marked improvements. For example, when their model is trained on ImageNet and tested on ImageNet-Rendition (a dataset of cartoons, origami, paintings, toys, etc.), the model accuracy increases from 33.0% to 44.8%.

Paper

Other Recent Robustness Papers

Improving test-time adaptation to distribution shift using data augmentation.

Augmenting data by mixing discrete cosine transform image encodings.

Teaching models to reject adversarial examples when they are unsure of the correct class.

TruthfulQA: Measuring How Models Mimic Human FalsehoodsModels trained to predict the next token are incentivized to repeat common misconceptions.

A new benchmark shows that GPT-3 imitates human misconceptions. In fact, larger models more frequently repeat misconceptions, so simply training more capable models may make the problem worse. For example, GPT-J with 6 billion parameters is 17% worse on this benchmark than a model with 0.125 billion parameters. This demonstrates that simple objectives can inadvertently incentivize models to be misaligned and repeat misinformation. To make models outputs truthful, we will need to find ways to counteract this new failure mode.

Paper

Other Recent Monitoring Papers

An expanded report towards building truthful and honest models.

Using an ensemble of one-class classifiers to create an out-of-distribution detector.

Provable performance guarantees for out-of-distribution detection.

Synthesizing outliers is becoming increasingly useful for detecting real anomalies.

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned ModelsAs networks become larger, they can more aggressively optimize proxies and reduce performance of the true objective.

Real-world constraints often require implementing rough proxies instead of our true objectives. However, as models become more capable, they can exploit faults in the proxy and undermine performance, a failure mode called proxy gaming. This paper finds that proxy gaming occurs in multiple environments including a traffic control environment, COVID response simulator, Atari Riverraid, and a simulated controller for blood glucose levels. To mitigate proxy gaming, they use anomaly detection to detect models engaging in proxy gaming.

Paper

Other Recent Alignment Papers

A paper studying how models may be incentivized to influence users.

Safe exploration in 3D environments.

Recent External Safety Papers

A thorough analysis of security vulnerabilities generated by Github Copilot.

An ML system for improved decision making.

Other News

The NSF has a new call for proposals. Among other topics, they intend to fund Trustworthy AI (which overlaps with many ML Safety topics), AI for Decision Making, and Intelligent Agents for Next-Generation Cybersecurity (the latter two are relevant for External Safety).

Discuss

### Causal vs Predictive Models, and the Causal Taboo

18 октября, 2021 - 18:05
Published on October 18, 2021 3:05 PM GMT

(I wrote this post in April 2020 for a non-LW audience)

Causation is pretty cool. Even cooler than causation, causal models! If you haven't heard the news, the past few decades have produced big leaps in understanding causality and how to reason about it. There's also been great descriptive work on how humans already intuitively deal with causality. Causality is so baked into the human mind that causal relationships can often be experienced at the perceptual level, even before any higher level reasoning systems can act. We're very good at spotting causal relationships when they're present, so good that we sometimes even detect them when they aren't there :)

To get an understanding of the difference between causal models and predictive models we're gonna use the slogan "Correlation does not equal causation" as our entry.

To be fair, that graph isn't really even a good example. That's more "just because lines sorta match up on a graph, doesn't mean you should expect them to continue to math up." I'd totally bet that these variables wouldn't even be correlated if you picked a different time span.

Here's a better example of correlation not equaling causation (based on a true story, but simplified to make a point):

Your new study finds that people who listened to Mozart as a kid have higher SAT scores. Mozart makes you smarter! The meaning inherent in that claims is "If I intervene by playing Mozart for a kid, they will become smarter than they otherwise would have."

What could go wrong? Well it turns out that the actual causal graph looks more like this:

Different story. If you know someone listened to Mozart as a kid, it is still 100% legit an accurate to predict higher SAT scores, but now it's also clear that intervening on the Mozart variable won't affect one's SAT. Mozart is useful for predicting SAT because they are caused by the same variable. Mozart is evidence that their family is wealthy, and that causally affects SAT scores. But if you control for wealth, and the Mozart effect goes away.

(Also for completion, the causal graph probs looks more like this):

Moral of the story: if you do stats well, you'll have a good model that you can use to make robust predictions, but it can't inform intervention unless you have a causal model relating the variables.

So that's one nugget of wisdom that can be taken from "Correlation does not equal causation". But there's some interesting historical baggage with the sentiment expressed by that slogan. It seems like the early pioneers who coalesced statistics into it's modern form (Ronald Fischer, Karl Pearson) were so pissed by people incorrectly inferring causaility, and by their own inability to formalize causality, they said "Fuck it! We are only studying correlation, there is NO SUCH THING as causality, and no one gets to use that disgusting word in my house!" I won't get into the weeds, but The Book of Why covers this section of history, and this book review offers some alternative stories.

Whatever the reason for this taboo, it seems clear that there was a taboo. Explicitly talking about causality or trying to develop formal models and theories about causality just wasn't something one did in polite statistical society (and statistics had gained a lot of clout for it's impressive predictive powers, so the force of this taboo slightly leaked into other sciences).

Prediction is cool, but intervention is even cooler. So even though the formal doctrine banned causality, that didn't stop statisticians from thinking causally (just like the Behaviorists, who had a similar pact to just pretend like consciousness didn't exist, never stopped being conscious). They even developed randomized control trials, which are very useful for determining causal relationships. That must have been an interesting tension to live with. Intensely needing something to do anything useful, while denying that said something exists in the first place. Good thing you and I don't do that with anything! Ha! Ha...

Long story short, in the year 2020, causal models and causal discovery are rich developed fields that are gaining momentum, with plenty of interesting stuff still to figure out.

Discuss

### Predictive Categories Make Bad Causal Variables

18 октября, 2021 - 18:02
Published on October 18, 2021 3:02 PM GMT

(this was written in April 2020 and I only just now realized I never posted it to LW)

This post is going to explore the consequences of different choices you can make when thinking about things causally. Shout out to johnswentworth for first seeding in my head this sort of investigation.

One mistake people are known to make is to vastly underestimate the causal factors behind a variable. Scott writes about this tendency in genetics:

What happens if your baby doesn’t have the gene for intelligence? Can they still succeed? [...] By the early 2000s, the American Psychological Association was a little more cautious, was saying intelligence might be linked to “dozens – if not hundreds” of genes. [...] The most recent estimate for how many genes are involved in complex traits like height or intelligence is approximately “all of them” – by the latest count, about twenty thousand.

Probably not too surprising. Everyone wants "The One Thing" that explains it all, but normally its the case that "These 35,000 Things" explain it all. The Folk Theory of Essences might be the most egregious example of people inferring a mono-causal relationship when reality is vastly poly-causal. George Lakoff (the metaphors and embodied cognition guy) explains:

The Folk Theory of Essences is commonplace, in this culture and other cultures around the world. According to that folk theory, everything has an essence that makes it the kind of thing it is. An essence is a collection of natural properties that inheres in whatever it is the essence of. Since natural properties are natural phenomena, natural properties (essences) can be seen as causes of the natural behavior of things. For example, it is a natural property of trees that they are made of wood. Trees have natural behaviors: They bend in the wind and they can burn. That natural property of trees-being made of wood (which is part of a tree's "essence")-is therefore conceptualized metaphorically as a cause of the bending and burning behavior of trees. Aristotle called this the material cause.

As a result, the Folk Theory of Essences has a part that is causal. We will state it as follows: Every thing has an essence that inheres in it and that makes it the kind of thing it is. The essence of each thing is the cause of that thing's natural behavior.

Thinking in terms of essences is very common. It seems to be how a lot of people think about things like personality or disposition. "Of course he lied to you, he's a crook" "I know it was risky and spontaneous, but I'm an ENTJ, so yeah."

My first reflex is to point out that your behavior is caused by more than your personality. Environmental contexts have huge effects on the actions people make. Old news. I want to look at the problems that pop up when you even consider personality as a causal variable in the first place

Implicit/Emergent Variables

Let's think about modeling the weather in a given region, and how the idea of climate factors into. A simple way to model this might be with the below graph:

Certain geographic factors determine the climate, and the climate determines the weather. Boom, done. High level abstraction that let's us model stuff.

Let's see what happens when we switch perspectives. If we zoom in to a more concrete, less abstract model, where the weather is a result of things like air pressure, temperature, and air density, all affecting each other in complex ways, there is no "climate variable" present. A given region exhibits regularities in its weather over time. We see similarities between the regularities in different regions. We develop labels for different clusters of regularities. We still have a sense of what geographic features lead to what sorts of regularities in weather, but in our best concrete models of weather there is no explicit climate variable.

What are the repercussions of using one model vs the other? It seems like they could both be used to make fine predictions. The weirdness happens when we remember we're thinking causally. Remember, the whole point of causal reasoning is to know what will happen if you intervene. You imagine "manually setting" causal variables to different values and see what happens. But what does this "manual setting" of variables look like?

In our graph from last post:

all the variables are ones that I have some idea on how to manually set. I can play Mozart for a kid. I can give someone's family more money. I can get College Board to give you fake SAT scores. But what would it mean to intervene on the climate node?

We know that no single factor controls the climate. "Desert" and "rain-forest" are just labels for types or regularities in a weather system. Since climate is an emergent feature, "intervening on climate" means intervening on a bunch of geographic variables. The previous graph leads me to erroneously conclude that I could somehow tweak the climate without having to change the underlying geography, and that's not possible. The only way to salvage this graph is to put a bunch of additional arrows in, representing how "changing climate" necessitates change in geography.

Contrast this with another example. We're looking at the software of a rocket, and for some reason the developer chose to hardcode the value into every location where they needed the value of the gravitational constant. What happens if we model the software as having a causal variable for g? Like climate, this g is not explicit; it's implicit. There's no global variable that can be toggled to control g. But unlike climate, this g isn't really an emergent feature. The fact that the software acts as if the gravitational constant is is not an complex emergent property of various systems interacting. It's because you hardcoded everywhere.

If we wanted to model this software, we could include a causal variable for every instance of , but we could just as easily lump that all into one variable. Our model would basically give the same answer to any intervention question. Yeah, it's more of a pain to find and replace every hardcoded value, but it's still the same sort of causal intervention that leaves the rest of the system intact. Even though g is an implicit variable, it's much more amenable to being modeled as an explicit variable at a higher level of abstraction.

Causal Variables and Predictive Categories

A few times I've told a story that goes like this: observe that a system has regularities in it's behavior, see other systems with similar clusters of regularity, develop a label to signify "System that has been seen to exhibit Type X regularities."

Previously I was calling these "emergent features", but now I want to frame them as predictive categories, mostly to emphasize the pitfalls of thinking of them as causal variables. For ease, I'll be talking about it as a dichotomy, but you can really think of it as a spectrum, where a property slides from being relatively easy to isolate and intervene on while leaving the rest of the system intact (g in the code), all the way up to complete interdependent chaos (more like climate).

A problem we already spotted; thinking of a predictive category (like climate) as a causal variable can lead you to think that you can intervene on climate in isolation from the rest of the system.

But there's an even deeper problem. Think back to personality types. It's probably not the case that there's an easily isolated "personality" variable in humans. But it is possible for behavior to have regularities that fall into similar clusters, allowing for "personality types" to have predictive power. Focus on what's happening here. When you judge a person's personality, you observe their behavior and make predictions of future behavior. When you take a personality quiz, you tell the quiz how you behave and it tells you how you will continue to behave. The decision flow in your head looks something like this (but with more behavior variables):

All that's happening is you predict behavior you've already seen, and other behavior that has been know to be in the same "cluster" as the behavior you've already seen. This model is a valid predictive model (results will vary based on how good your pattern recognition is) but gives weird causal answers. What causes your behavior? Your personality. What causes your personality? Your behavior.

Now, it's not against the rules of causality for things to cause each other, that's what control theory is all about (play with negative feedback loops here!). But it doesn't work with predictive categories[^1]. Knowing what personality is, we can expand "Your personality causes the regularities in your behavior" to "Your regularities in your behavior cause the regularities in your behavior." There is no causal/explanatory content. At best, it's a tautology that doesn't tell you anything.

This is the difference between personality and climate. Both are predictive categories, but with climate we had a decent understanding of what variables we might need to alter to produce a "desert" pattern or a "rain-forest" pattern. How the hell would you change someone from an ENTP pattern to an ISFJ pattern? Even ignoring the difficulties of invasive brain surgery, I don't think anyone has any idea on how you would reshape the guts of a human mind to change it to another personality cluster.

Thinking of personality as a causal node will lead you to believe you have an understanding that you don't have. Since you're already conflating a predictive model for a causal one, you might even build a theory of intervention where you can fiddle with downstream behavior to change the predictive category (this sort of thinking we'll explore more in later posts).

To recap: if you treat a predictive category as a causal variable, you have a good chance at misleading yourself on your ability to:

• Intervene on said variable in isolation from the rest of the system.
• Perform an intervention that shifts you to another predictive category cluster.
Humans and Essences

Finally we circle back to essences. You can probably already put together the pieces. Thinking with essences is basically trying to use predictive categories as causal nodes which are the source of all of an entities behavior. This can work fine for predictive purposes, but leads to mishaps when thinking causally.

Why can it be so easy to think in terms of essences? Here's my theory. As already noted, our brains are doing causal learning all the time. The more "guts" of a system you are exposed to, the easier it is to learn true causal relationships. In cases where the guts are hidden and you only interact with a system as a black-box (can't peer into people's minds), you have to rely on other faculties. Your mind is still great at pattern recognition, and predictive categories get used a lot more.

Now all that needs to happen is for you to conflate this cognition you use to predict, for cognition that represents a causal model. Eliezer describes it in "Say not complexity":

In an eyeblink it happens: putting a non-controlling causal node behind something mysterious, a causal node that feels like an explanation but isn’t. The mistake takes place below the level of words. It requires no special character flaw; it is how human beings think by default, how they have thought since the ancient times.

An important additional point is to address why this easy to make mistake doesn't get corrected (I make mistakes in arithmetic all the time, but I fix them). The key piece of this not getting corrected is the inaccessibility of the guts of the system. When you think of the essences of people's personalities, you don't get to see inside their heads. When Aristotle posited the "essence of trees" he didn't have the tools to look into the tree's cells. People can do good causal reasoning, but when the guts are hidden and you've got no way to intervene on them, you can posit crazy incorrect causal relationships all day and never get corrected by your experience.

Quick Summary
• Properties of a system can be encoded implicitly instead of explicitly.
• The more a property is the result of complex interactions within a system, the more likely it is to be a predictive category instead of a useful causal variable.
• When you treat a PC as a CV, you invite yourself to overestimate the ease of intervening on the variable in isolation from the rest of the system, and to feel like you know how to coherently alter the value of the variable even when you don't.
• The less exposed you are to the guts of a system, the easier it is to treat a predictive model as a causal one and never get corrected.

[^1] The category is a feature of your mind. For it to exert cause on the original system, it would have to be through the fact that you using this category caused you to act on the system in a certain way. When might you see that happen?

Discuss

### Epistemic Strategies of Selection Theorems

18 октября, 2021 - 11:57
Published on October 18, 2021 8:57 AM GMT

Introduction: Epistemic Strategies Redux

This post examines the epistemic strategies of John Wentworth’s selection theorem posts.

(If you want to skim this post, just read the Summary subsection that display the different epistemic strategies as design patterns)

I introduced the concept in a recent post, but didn’t define them except as the “ways of producing” knowledge that are used in a piece of research. If we consider a post or paper as a computer program outputting (producing) knowledge about alignment, epistemic strategies are the underlying algorithm or, even more abstractly, the design patterns.

An example of epistemic strategy, common in natural sciences (and beyond), is

• Look at the data
• Find a good explanation
• Predict new things with that explanation
• Get new data for checking your prediction

More than just laying out some abstract recipe, analysis serves to understand how each step is done, whether that makes sense, and how each step (and the whole strategy) might fail. Just like a design pattern or an algorithm, it matters tremendously to know when to apply it and when to avoid it as well as subtleties to be aware of.

Laying this underlying structure bare matters in three ways:

• It clarifies the research’s purpose and value for newcomers and researchers from other fields, with minimal assumptions of shared approaches.
• Just like a programmer switching to a new domain of problems will get up to speed faster and more reliably if they get access to the patterns/algorithms/tricks used in their new domain.
• It focuses feedback and criticism on the most important parts of the idea/proposition/argument.
• Issues with an algorithm more often focus on the point of it instead of the details,whereas issues with an implementation of that algorithm can be as much about typos, optimization tricks and bad structure than about the actual core (the algorithm.
• It builds a library of such strategies for alignment in particular, a cookbook newcomers and senior researchers alike can browse for inspiration or a take on some new post/paper they don’t grok.

Now, before starting, I need to point out that the selection theorems posts don’t present research results; they present epistemic strategies (the eponymous theorems). Does that mean my job has already been done? Not exactly: John’s posts do present that epistemic strategy, but not in all the ways I want to stress out. John is also trying to fill in a lot of concrete details and to convince people that selection theorems are a nice thing to research, which I don’t have to do. Instead, you can see this post as distilling the structure of selection theorems and interrogating them further as ways of producing knowledge.

(I use the word “agent” to stay coherent with John, but nothing in the epistemic strategy itself requires agency, and so finding the idea of agents confusing shouldn’t be an issue for reading this post)

Thanks to John Wentworth for feedback on a draft of this post.

Characterizing Selection Theorems

Selection theorems are theorems. Obviously. But what sort of theorems? What are they trying to find about the world?

John summarizes the whole class of results in the following way:

Roughly speaking, a Selection Theorem tells us something about what agent type signatures will be selected for (by e.g. natural selection or ML training or economic profitability) in some broad class of environments.

This gives us three components of a selection theorem: the selection pressure, the class of environments considered and the constraint on the agent (what John calls the “type signatures”). Let’s get into each, looking for what can fill the corresponding hole in the general selection theorem.

Selection

A selection theorem is first and foremost about selection. Not just selection mechanisms (low-level processes like natural selection) but also selection criteria (abstract conditions like no Dutch-booking). The former state how selection happens, whereas the other just characterize the sort of things that will be selected.

One of the differences is that a selection mechanism implies a selection criterion, either implicitly (natural selection) or explicitly (ML training with an actual loss function); whereas a selection criterion doesn’t necessarily come with a mechanism.

Still, both mechanisms and criteria come in a wide variety -- what makes a good one for a selection theorem? Making the selection theorem applicable to the real world situation we care about. The next section focuses on this topic of application, but in summary: mechanisms must fit actual selection processes in the situation, whereas criteria must come with an explanation of why they would be instantiated (possibly a corresponding selection mechanism, but not necessarily).

It’s also less obvious what makes a “good” criterion, because of the risk to assume the constraint we want to show in the selection criterion itself.

Environments

I find John’s formulation above unfortunate, because it doesn’t stress enough how the “broad class of environment” is part of the hypothesis of a selection theorem, not the conclusion. The intuition here is that we need enough variety to instantiate the selection pressure or criterion. Selection let’s you force the agent’s hand, but only if you can instantiate the situations you need.

For a selection mechanism, this amounts to containing the sort of situations where the mechanism will push in the right direction and be strong enough (for example predation pushing natural selection forward). For a selection criterion, it is about including the situations that take advantage of every suboptimality in the agent (like the exploitative bets punishing suboptimality in no Dutch-Booking)

Note though that while a broader class of environments might be necessary for proving the theorems, it makes applying it more difficult by putting more conditions on the environments in the real world setting. There is thus a trade-off between making it possible to prove the theorem (more environments) and making it possible to apply it (less environments). We thus want as small a set of environments as possible while still being large enough to leverage the selection.

Constraint on agents

In the original post, John takes pains to split agents’ type signatures into different components and to explain how they interact with each other. At the level we’re seeing stuff though, we only need to understand that type signatures are necessary conditions on the agents coming from the selection: if an agent is to be selected, it must do X (or do X with high probability).

What sort of conditions do selection theorems show? Here we have a discrepancy between what selection theorems historically prove and what John wants to get out of them. Existing selection theorems only prove behavioral necessary conditions: you must act like this (as in coherence theorems) or you must be able to do that (as in the Gooder Regulator theorem). On the other hand, what we truly want are structural necessary conditions — for example “you must have a separate world model with this interface and these properties”. John’s third post on selection theorems is all about how he wants that.

Indeed, structural constraints tell you not only that the system must solve the problem, but how it will do so. Alignment just becomes easier if we have knowledge of the internal structure of the system: we can make more pointed predictions about how it might be unaligned; we might use this structure for more concrete alignment schemes. Fundamentally, structural constraints give us back some of the guarantees of the main epistemic strategies of Science and Engineering that get lost in alignment: we don’t have the technology yet, but we have some ideas of how it will work.

I’ll go into more detail about proving structural constraints in the next section, but for the moment just note that this is the sort of thing we want.

Summary

Selection theorems thus have the following structure:

• Hypotheses
• (Selection pressure) Some means of selection, either a mechanism or a criterion.
• (Environments) A class of environments broad enough to instantiate the selection pressure in needed way, but small enough to still apply the theorem to real world settings.
• Conclusion
• (Necessary Condition on Agents) Some property (ideally structural but maybe behavioral) that is guaranteed for all selected agents, or at least with high probability.
Proving Selection TheoremsBehavioral constraints

Existing selection theorems only prove behavioral constraints — that is, they only show that the agents must be behaviorally equivalent to a specific class (like EU maximizers in coherence theorems) or that they must be able solve a specific problem (remembering all relevant data in the Gooder Regulator theorem).

How to prove selection theorems for behavioral constraints? Looking at the existing theorems, the first thing to notice is that they tend to use selection criteria. It makes sense, as they tend to be proved backwards: looking at the necessary condition on agents, what criterion selects only agents behaving like this?

It doesn’t mean such theorems are trivial or useless, just that they tell us which criterion selects for the necessary condition, not what is selected by some selection pressure.

Structural constraints

Here instead of criteria, mechanisms are favored. This is mostly because we want to show that some process (natural selection and/or ML training) leads to structural constraints, not find criteria for structural constraints.

Note that we should expect any mechanism to find some good ad-hoc agent without the structural properties; selection theorems for structural constraints can thus only give probabilistic guarantees. They say “out of the agents favored by this selection mechanism, most/almost all will have these structural properties”.

Here are some epistemic strategies to argue that the typical agents selected by a selection theorem on behavior alone should in expectation have additional structural constraints. The list isn’t exhaustive, and I expect these strategies to be combined when actually arguing for such structural constraints.

• (Agents with these structural constraints are easier to find) Especially with a selection mechanism, we can argue for properties of the selected agents that are easier to find.
For example, John argues that robust and broad optima are easier to find and retain through mechanisms for selection like gradient descent or natural selection, and proposes that these optima might correspond mostly to agents with modular structures.
• (Agents with structural constraints are a majority) If we can show that most of the selected-for agents have these structural constraints, that is some evidence that we should expect that structure. Not as strong as with an explanation of why these would be favored though.
Note that this applies both to mechanisms and criteria.
• (Agents with structural constraints are easier to sample) I already described this epistemic strategy, in relation with Alex Turner’s work on Convergent Subgoals and a comparison to Smoothed Analysis. Basically, if one can show that the agents without structural properties are so rare that they correspond to very steep high peaks in a mostly flat landscape, all but very few sampling of selected-for agents will end up satisfying the structural properties.
• (Proposed sampling gets structural constraints with high-probability) If we can propose a sampling method for agents, argue that it indeed fits with how the selection pressure eventually samples (like the proposal here for SGD), and show this sampling to find in expectation agents with the structural constraint, that’s a very strong argument for assuming this structure.
Summary

Proving selection theorems use the following epistemic strategies:

• Proving behavioral selection theorem
• Choose a necessary condition to investigate.
• Find a selection criterion that should favor the necessary condition.
• Prove the theorem.
• Proving structural selection theorem
• Choose a selection mechanism to investigate.
• Find a structural constraint that should be favored by the mechanism.
• Prove the theorem.
• Show that agents with these structural constraints are easier to find.
• Show that agents with structural constraints are a majority.
• Show that agents with structural constraints are easier to sample.
• Propose a sampling of agents and show it results in structural constraints with high-probability.
Applying Selection Theorems

Even pure mathematicians don’t prove theorems only for the joy of the proof: the value of a theorem often comes from what it shows and where it can be applied. The same holds in alignment, with the additional difficulty that we want to apply it to real world systems and situations, not only to other abstractions. This means we need to understand when we can apply selection theorems and what we can learn from that application.

Requirements of selection theorems

First thing first: selection theorems require the existence of selection. Once again quite obvious, but it becomes more interesting if we dig into the subtleties.

How to argue for the existence of selection depends on whether the theorem uses a mechanism or a criterion.

• (Mechanism) The question is whether the mechanism actually happens in the real world application. Answering this question can go from trivial (we know ML training happens because we’re the one implementing it) to yielding epistemic strategies used for showing selection happens (like the arguments for natural selection).
• (Criterion) An additional difficulty with a criterion is that we need to justify that selection along this criterion indeed happens. That doesn’t necessarily mean providing a full selection mechanism, but we need at least reasons for why this would happen.
Most selection theorems using criteria (like coherence theorems) propose a high-level selection mechanism for this purpose.

The other requirement lies on environments. Not only do we need the variety of environments over which selection is taking place, but environments also need to fit the mold assumed in selection theorems. Coherence theorems for example require that bets can be defined in the environments with the required properties, and that the space of bets considered contains the dutch-booking strategies for any suboptimal policy. The Gooder Regulator Theorem has more concrete requirements in terms of the underlying causal structure, and the same sort of variety constraint on the “tasks” that the agent has to solve.

Interpreting the application of selection theorems

Once we are confident the selection theorem applies in our concrete setting, we can reap its fruits. But what are those fruits? At first glance, they’re obvious: the necessary conditions stated in the theorem! Yet anyone who ever applied a theorem to a real world setting knows how perilous that task is.

How do you make sense of the necessary conditions in your setting for example? You need to find a way of grounding the constraints on agents you get out of the theorem.

This is where most applications of theorems to real world settings go wrong, in my opinion. Yet this is also the part I have the least to say about, because I just don’t have some nice epistemic strategy to check that some conclusions taken from applying a theorem to situation S actually make sense. I’ve seen people do that move, I’ve made it myself, but I don’t have a nice description of the underlying algorithm. So let’s flag that as an open problem for the time being.

Summary

Proving selection theorems use the following epistemic strategies:

• Checking that the selection theorem applies
• Check that the selection exists.
• For a mechanism, check that it fit with how selection happens.
• For a criterion, find a (possibly high-level) mechanism for why selection happens along these lines.
• Check that the environments fit the required structure.
• Check that the environments fit the required variety.
• Interpreting the theorem after applying it
• Open Epistemic Strategy Problem
• How to interpret a behavioral constraint?
• How to interpret a structural constraint?
• That we can model the agent coherently with that structure?
• That the agent implements that structure explicitly?
Breaking Selection Theorems

Last but not least, analyzing an epistemic strategy tells us where it can go wrong. The analogy to think here is of falsification: this is a standard and strong way of trying to break a scientific model. What does that look like for selection theorems?

Let’s use the summary design patterns of the previous section, and for each one, finding issues/criticisms/ways of breaking that step.

• Proving behavioral selection theorem
• Choose a necessary condition to investigate.
• Find a selection criterion that should favor the necessary condition.
• Find a counterexample (agent selected by criterion but not satisfying the necessary condition; or subset of enough agents to break probabilistic condition).
• Prove the theorem.
• Find an error in the proof.
• Proving structural selection theorem
• Choose a selection mechanism to investigate.
• Find a structural constraint that should be favored by the mechanism.
• Prove the theorem.
• Show that agents with these structural constraints are easier to find.
• Show that many agents without the structural constraints can be easily found by the selection pressure.
• Show that agents with structural constraints are a majority.
• Show that there isn’t a majority of selected-for agents with structural constraints.
• Show that agents with structural constraints are easier to sample.
• Argue that the set of selected-for agents is different that the one used in the work, and that for the actual set, sampling agents without structural constraints becomes simpler.
• Propose a sampling of agents and show it results in structural constraints with high-probability.
• Show that the proposed sampling disagrees with what the selection pressure actually finds (showing that the probabilities are different, or that one can sample agents that the other can’t).
• Checking that the selection theorem applies
• Check that the selection exists.
• For a mechanism, check that it fits with how selection happens.
• Show that the actual selection works differently than the mechanism described, and that these differences influence massively what is selected in the end.
• For a criterion, find a (possibly high-level) mechanism for why selection happens along these lines.
• Argue that the posited high-level selection mechanism for the criterion doesn’t exist or that it doesn’t push towards the criterion.
• Check that the environments fit the required structure.
• Show that the concrete environments don’t fit the constraints of the theorem.
• Check that the environments fit the required variety
• Show that the concrete environment lacks some situations that are needed to make the proof hold.
• Interpreting the theorem after applying it
• Open Epistemic Strategy Problem
• How to interpret a behavioral constraint?
• How to interpret a structural constraint?
• That we can model the agent coherently with that structure?
• That the agent implements that structure explicitly?.

Lastly, in addition to criticizing a specific application of the theorem, we might argue that the theorem cannot  be applied to the wanted setting, or that it doesn’t make sense to conclude what is wanted from it. This amounts to the points above, with the twist of arguing that it’s impossible instead of just breaking the argument at some joint.

This obviously fails to list all possible ways of critiquing a selection theorem and its application. You might have noted that I didn’t say anything about interpreting the necessary condition once the theorem is applied; indeed, without understanding the epistemic strategy involved, it’s harder to get to the core.

Still, any criticism and feedback along these lines would be directly useful to the researcher (John or someone else) proposing a new selection theorem and/or applying one. My claim is that using the design pattern above helps in providing feedback, by drawing attention to the most important parts of the epistemic strategies involved.

Discuss

### Humans are the universal economic bottleneck

18 октября, 2021 - 11:32
Published on October 18, 2021 8:32 AM GMT

There's this idea in computer science wherein the maximum theoretical speedup that can be acquired with an arbitrary number of processors is related to the percentage of the program which can be parallelized. If we have two segments of code that take the same amount of time to execute with one CPU core in which the first segment can't be parallelized at all and the second segment is perfectly parallelizable, we can only run the program twice as fast, no matter how many CPU cores we have.

There's a similar idea in economics. It seems like the most powerful and civilizationally relevant feature controlling the medium to long term change in the price of goods is the extent to which the production of that good can be decoupled from the expenditure of man-hours. Some economic activity isn't "parallelizable" using current technology- we can't practically make that activity much faster without building powerful substitutes for humans, technology which is (for now) mostly out of our reach...

For example, it turns out that moving stuff over land is not easy to decouple from human labor without self-driving cars. There are two methods of overland transportation worth noting here: Cars and Trains.

Due to the nature of our road infrastructure, there's a pretty clear upper bound on how efficient car-based transportation of goods can get. There are legal limits on the allowed speed and size of vehicles, so without self-driving tech, we can't change how many man-hours need to be spent per cubic meter per kilometer.

Train-based transportation has its own problems which limit its ability to dominate overland transportation. Namely, our train transportation network is incomplete in that goods must still be ferried to their final destination by cars, so overland transportation is only some % "parallelizable".

Using this we can predict that overseas transportation would be really efficient compared to overland transportation. I suspect that Nautical miles per cargo container per hour per person can be pushed extremely high using current technology, laws, port and canal infrastructure, etc.

And indeed this is true:

Be warned that overland freight for even short distances can often be almost as much as ocean freight for thousands of miles (I recently paid $2000 for a 20′ container to be shipped from Shanghai to Los Angeles and then$1100 for it to be shipped 30 miles from Los Angeles to San Moreno).

Our ability to recursively reinvest our production is most strongly limited by these required industries which are mostly bottlenecked by the number of humans and how long they're willing to work, neither of which are easy to manipulate and neither brings the sort of powerful prosperity that characterizes modernity.

EDIT: Someone commented that other factors are at play that makes cargo ships efficient. I do not disagree- this was just an example of the sort of weak estimate which could be made using this idea. I am interested in determining how important this effect is in the case of cargo ships, so I will do a short analysis.

Compare to semi-trucks which can carry roughly 40,000 kg of material. We will say these trucks move at 100km/h. The wage of a semi-truck driver in the U.S. is roughly $20, so combing we have40000kg∗100km$20.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-surd + .mjx-box {display: inline-flex} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor; overflow: visible} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} , so it costs $1 to move 200,000kg one kilometer- or rather, that is the wage component. The distance between Shanghai and Los Angeles is ~10,000km, and the limit weight of a 20ft shipping container is about 80,000kg. Assuming wage costs for ship crew are negligible (this should cancel out with previous generous estimates), we have 800,000,000kg * km for this trip for this container. If the same wage were required per kg per km as in the semi-truck case (note much is ignored here for the sake of approximation), this would cost an additional$4000. For comparison, according to this site, the cost of shipping 80,000kg from Shanghai to LA is $14000. Discuss ### Quick Look: Altitude and Child Development 18 октября, 2021 - 04:20 Published on October 18, 2021 1:20 AM GMT A client came to me to investigate the effect of high altitude on child development and has given me permission to share the results. This post bears the usual marks of preliminary client work: I focused on the aspects of the question they cared about the most, not necessarily my favorite or the most important in general. The investigation stops when the client no longer wants to pay for more, not when I’ve achieved a particular level of certainty I’m satisfied with. Etc. In this particular case they were satisfied with the answer after only a few hours, and I did not pursue beyond that. That out of the way: I investigated the impact of altitude on childhood outcomes, focusing on cognition. I ultimately focused mostly on effects visible at birth, because birth weight is such a hard to manipulate piece of data. What I found in < 3 hours of research is that altitude has an effect on birth weight that is very noticeable statistically, although the material impact is likely to be very small unless you are living in the Andes. Children gestated at higher altitudes have lower birth weights This seems to be generally supported by studies which are unusually rigorous for the field of fetal development. Even better, it’s supported in both South America (where higher altitudes correlate with lower income and lower density, and I suspect very different child-rearing practices) and Colorado (where the income relationship reverses and while I’m sure childhoods still differ somewhat, I suspect less so). The relationship also holds in Austria, which I know less about culturally but did produce the nicest graph. This is a big deal because until you reach truly ridiculous numbers, higher birth weight is correlated with every good thing, although there’s reason to believe a loss due to high altitude is less bad than a loss caused by most other causes, which I’ll discuss later. [Also for any of you wondering if this is caused by a decrease in gestation time: good question, the answer appears to be no.] Children raised at higher altitudes do worse on developmental tests There is a fair amount of data supporting this, and some even attempt to control for things like familiar wealth, prematurity, etc. I’m not convinced. The effects are modest, I expect families living at very high altitudes (typically rural) to be different in many ways from lower altitudes (typically urban) in ways that cause their children to score differently on tests without it making a meaningful impact on their life (and unlike birth weight, I didn’t find studies based in CO, where some trends reverse). Additionally, none of the studies looked specifically at children who were born at a lower altitude and moved, so some of the effects may be left over from the gestational effects discussed earlier. Hypoxia may not be your only problem I went into this primed to believe reduced oxygen consumption was the problem. However, there’s additional evidence that UV radiation, which rises with altitude, may also be a concern. UV radiation is higher in some areas for other reasons, which indeed seems to correlate with reductions in cognition. How much does this matter? (not much) Based on a very cursory look at graphs on GIS (to be clear: I didn’t even check the papers, and their axes were shoddily labeled), 100 grams of birth weight corresponds to 0.2 IQ points for full term babies. The studies consistently showed ~0.09 to 0.1 grams lower birth weight per meter of altitude. Studies showed this to be surprisingly linear; I’m skeptical and expect the reality to be more exponential or S shaped, but let’s use that rule of thumb for now. 0.1g/m means gestating in Denver rather than at sea level would shrink your baby by 170 grams (where 2500g-4500g is considered normal and healthy). If this was identical to other forms of fetal weight loss, which I don’t think it is, it would very roughly correspond to 0.35 IQ points lost. However, there’s reason to believe high-altitude fetal weight loss is less concerning than other forms. High altitude babies tend to have a higher brain mass percentage and are tall for their weight, suggesting they’ve prioritized growth amidst scarce resources rather than being straight out poisoned. So that small effect is even smaller than it first appears. There was also evidence out of Austria that higher altitude increased risk of SIDS, but that disappeared when babies slept on their backs, which is standard practice now. So gestating in Denver is definitely bad then? (No) There are a billion things influencing gestation and childhood outcomes, and this is looking at exactly one of them, for not very long. If you are making a decision please look at all the relevant factors, and then factor in the streetlight effect that there may be harder to measure things pointing in the other direction. Do not overweight the last thing I happened to read. In particular, Slime Mold Time Mold has some interesting data (which I haven’t verified but am hoping to at least ESC the series) that suggests higher altitudes within the US have fewer environmental contaminants, which you would expect to have all sorts of good effects. Full notes available here. Thanks to anonymous client for commissioning this research and Miranda Dixon-Luinenburg for copyediting. Discuss ### Your Time Might Be More Valuable Than You Think 18 октября, 2021 - 03:55 Published on October 18, 2021 12:55 AM GMT Summary • People often seem to implicitly value their time at the amount they can convert hours to dollars given their current skills. • However, the value of saving the marginal hour today is to increase the total number of one's working hours by one, resulting in a new hour at the end of one's career, not a new hour at their current skill level. • This suggests that people who expect their time to become valuable in the future must think their time is approximately just as valuable now, because saving time now gets them to the point where their time is valuable faster and gives them more of such time. • This analysis is complicated by various temporal dependencies (e.g. time discounting) that push the value of the current hour up or down compared to the value of the marginal hour at the end of one's career. • Under such a view, finding promising young altruists and speeding up their careers represents a significant value add. Intro Many people in my social circles have an amount they "value their time." Roughly speaking, if someone values their time at$50/hr, they should be willing to pay $50 to save an hour of time, or be paid$50 to do work that has negligible non-monetary value. Knowing this value can provide a simple decision rule for deciding which opportunities to trade money for time it's efficient for you to take. I will argue that a naive perspective on time evaluations generally results in an underestimate. This analysis suggests that altruistic actors with large amounts of money giving or lending money to young, resource-poor altruists might produce large amounts of altruistic good per dollar. I will analyze the situation mostly in terms of wages as expressed in dollars; however, readers might want to substitute "altruistic impact" instead. I will begin by analyzing a simplified situation, adding more nuance later.

The value of your time is the value of the marginal hour at the end of your career

If I currently have a job that lets me convert one hour of time into $50 dollars, then it's clear that I should take all time-saving opportunities at less than$50 dollars. (Note that this doesn't mean that I should pay $50 to save an hour of furniture assembly. Furniture assembly might be enjoyable, teach me valuable skills, etc.) However, this assumes that the benefits I receive from my job are entirely monetary. For most jobs, this will not be the case. If one is a software engineer, then much of the benefit of 1 hour of working as a software engineer will be the skills and experience gained during that hour. To be more specific, the hourly rate that a software engineer commands depends on their skill, which depends on training/experience in turn. Thus an hour of software engineering might increase expected future compensation by more than$50 (in fact, under plausible assumptions, this will be the primary benefit of the early part of most careers.)

However, as stated above, this suggests that the marginal hour at the present is worth w(t0), your current wage. This is not what actually happens when you save one hour at the present. What actually happens is that your total earnings of your career will now be ∫T+1t0w(t)dt for a difference of w(T) instead of w(t0). Since one's expected wage at the end of a career is likely substantially higher than ones current wage (especially for people at the beginning of their careers), treating the value of one's time as w(t0) instead of w(T) leads to an underestimate by w(T)−w(t0).

For example, suppose that one is a quantitative trader. They currently earn $100/hr. However, with 20,000 hours (10 years, assuming 2000 working hours a year) of experience, they expect to earn$1000/hr. If they have no time-discount rate on money, then they should be willing to pay up to $1000 to save an hour of time presently, despite the fact that they will be net down$900 if they use that time to do work. Another way of seeing this is that saving an hour of time for your present self is in some sense the same thing as saving an hour of time for your future self, because it causes the future to arrive one hour earlier and be one hour longer. Thus, if you would be willing to trade an hour for $1000 in the future, you should also be willing to do so now. This also suggests that the returns to working twice as much results in much more than twice the value produced. Naively, a 160,000 hour career produces the same value as two 80,000 hour careers. However, in reality, one of those careers is going to start with 80,000 hours of experience! This doesn't account for a lot of relative factors (being faster than competitors can produce much higher amounts of value) or aging-out effects like getting worse at working as you work more. A corollary is that burning out for a year is a disaster, because it's equivalent to losing a final career year. Similarly, vacations and other such leisure activities have larger costs than one might have naively expected, since they delay career growth and shorten careers. For example, if someone who could have had a 40 year career burns out for a year, their career is now 39 years and is missing the year where they would have had 39 years of experience. Temporal Dependence One key factor missing in the above analysis is a temporal dependence on the value of wages. (The substitution of wages for altruistic impact is going to break down slightly and depend on complicated factors like the flow-through effects of altruism and whether standard investment returns are higher than altruistic flow-through effects. See Flow-Through Effects of Innovation Through the Ages and Giving Now vs. Later for a more nuanced discussion.) The most obvious form of temporal dependence is a monetary discount rate controlled by the ability to turn money now into more money later via standard investments. Such a discount rate suggests that our theoretical quantitative trader discussed above should not be willing to spend$1000 to save an hour of time at the present day, but rather spend an amount that would be equivalent to $1000 after 10 years of investment (approximately$500 at 7% yearly returns). I could write an equation expressing this, but I don't think it would lend much clarity.

Less standard but more accurate analyses would incorporate the relative differences in the value of money over time for your particular goals. For instance, it might be that the altruistic discount rate on dollars is much higher than the standard discount rate because there are altruistic opportunities available now that won't be available later, even if you had double the money. Another salient example is effective altruism movement building (meta-EA), which might get most of its value early on. One way to model this is that instead of producing value directly, people in meta-EA save other people's time (by getting them into the movement earlier), enabling them to produce more value later. If you think, for example, that this century is particularly important, then saving an early career altruistic professional 1 year of time in 2090 is going to get you the marginal year of someone with ~10 years of experience, compared to saving such a person 1 year in 2080, which gets the marginal year with ~20 years of experience. Depending on how quickly you think the value of someone's work goes up with respect to experience, then this might suggest large discount rates.

As another example, people working in AI Alignment (like me) might think that most valuable alignment work is going to be done in the ~10 years preceding transformative AI (TAI). If you think this date is about 2055 (see Holden's summary of Ajeya's Forecasting TAI from Biological Anchors), then the most important thing is to maximize your abilities as a researcher from 2045-2055. (It's possible that you should be making different bets, e.g. if you think you have more influence in worlds where TAI is sooner.) Since I'll probably still be working in 2055, saving a marginal year of time today gives me one extra year of research experience during the decade preceding TAI, but not any extra marginal years during that decade. (This does suggest that saving time during that decade is very valuable, though.) Of course, I am not modeling various effects that current research has on things like field building, which potentially dominates the value of my current work.

Actionables

This analysis suggests that people with the potential to earn high salaries/have high altruistic impact have high time value, not because they can produce useful work currently, but because it will get them to where they eventually will end up faster. Provided this holds qualitatively, it suggests a couple of things:

• Care about the value of your time more and try to aggressively take opportunities to save it or spend it more effectively, even if this doesn't make that much sense in terms of the value you think you can currently generate.
• This might mean that you should take out loans and such, so you have resources. If your expected future earnings are high, then things like hiring tutors to graduate school faster are likely worth the interest on the loan.
• What you spend your free time doing actually kind of matters. Developing some skill one year faster increases the amount of value you produce on the margin by quite a bit.
• Standard advice saying that young people have time to explore potential career options should be balanced against the cost of becoming less awesome in that particular career option because too much time was spent exploring.
• For example, if someone is potentially a promising AI Aligner, and they take a year off college to travel the world and see the sights, this decreases the amount of research experience they have during the period around TAI by a year.
• Exploration is still probably a good idea, but it should be traded off not against the value one would have produced directly, but rather the marginal increase in value that would have resulted from the increased growth/experience if that time wasn't spent exploring.
• Finding promising young people and using large amounts of resources to speed up their careers probably has pretty good altruistic returns.
• (If you think you're such a person and could benefit from additional resources, feel free to send me an email and I'll see what I can do.)

Discuss

### The Colonization of Cults, Nonprofit Organizations, and Society

18 октября, 2021 - 01:02
Published on October 17, 2021 10:02 PM GMT

Over the past 8+ years of nonprofit experience and during a brief stint of training with a high demand group focused on meditation and leadership development (The Monastic Academy) I have observed how patterns and ideologies related to the complex socio-emotional and historical contexts of American culture and colonization show up, again and again, both within the broader systematic issues nonprofits exist to address as well as within organizations themselves.

Recently while reviewing a list of characteristics and patterns common in “cult” dynamics I recognized that I was also looking at a list that describes colonization. According to a simple google search, colonization is the action or process of settling among and establishing control and domination over the indigenous people of an area. Historically, global colonization has often targeted and disproportionality affected many communities of color including the genocide of indigenous peoples, forced assimilation into cultural and religious practices, loss of language and culture, taking of indigenous lands, the enslavement of Africans and other peoples, forced separation and abuses of indigenous children in boarding schools, etc. Before then many groups within Europe had their own history of invasion, conquest, and colonization (i.e. spread of the Roman empire, English clearing of the Scottish Highlands.) These practices and the history of colonization has left a deep psychological, physical, emotional, and spiritual imprint on people from all walks of life. Unfortunately, these unconscious and conscious patterns/attitudes inherent in “colonization” still show up throughout all levels of our society, perpetuating harm and inequity, and are often deeply embedded within our frameworks for and understanding of community, leadership, institutional and organizational management. This makes it critically important to be intentional about recognizing and addressing unhealthy and dysfunctional patterns of behavior, structures, and practices that perpetuate harm directly and indirectly within our communities and organizations.

On the far end of this spectrum, we see high-demand groups, commonly known as “cults”.  Many of these groups operate under a 501c3 nonprofit status and under the guise of having a mission to bring transformative change or to be of service to the world. However, their outward-facing mission and values often prove to be incongruent with the internal narrative and actual impacts of the organization. Perhaps these groups are not as separate from the dominant culture as we might like to think but are in fact intense microcosms in which particular underlying ideologies, structures, and behaviors are taken to an extreme. Common characteristics, ideologies, and patterns within cults include but are not limited to recruiting of “elite or special ones”; we are the chosen ones; we are going to save the world; we have the right to have and exercise power over others because we are better in x,y, or z ways; unlimited expansion (Manifest Destiny, anyone?); use of religion/spirituality and power to control people and governments; dominated group submits to the will of the dominator; hierarchical and authoritarian (often patriarchal) styles of leadership, breaking down of ones personal and cultural identity and replacing it with a new cult dogma and identity, abuses of power and lack of accountability for those abuses; distorted and disempowered relationships between feminine/masculine energies and persons; disconnect/distrust of your own body and emotions; unhealthy relationship to resources (i.e. money, land) and resource extraction (i.e. unethical fundraising practices, illegal activities), ect]. Involvement in and hierarchies within these groups are often but not always reflected along lines of class, gender, and race reflected in the broader society as "cults" aka high demand groups often target people with money and greater social influence.

At the same time, "cults" themselves are likely a long-term cultural byproduct of colonization that has left many people rootless, with intergenerational trauma, experiencing "the loss of the village", with inadequate socio-emotional support networks, and a lack of cultural identity and connection to the cultures their ancestors came from. These impacts also include many people of European descent. It is important to acknowledge that capitalism has played a significant role in the breakdown of and the “loss of the village”.  Many people today, especially young people are hungry for a sense of cultural identity, belonging, initiation, community, guidance, and mentorship, searching for solutions to societal and environmental breakdowns, and a need for shared purpose and meaning that is lacking in the broader culture. Throw in a major loss or life transition or past childhood trauma without the support of a "village" and people are incredibly vulnerable to charismatic leaders who more or less promise to give them everything they have been looking for at a "price.” This price often being their agency, their power, their silence, access to their resources (money, sex, social influence) and their complicity in perpetuating harmful power structures and dynamics.

Many nonprofits and companies that do not fit the defining characteristics of a "cult" have also been guilty of perpetuating these systems, patterns, and practices that further disempower marginalized groups, individuals, and local communities (especially poor and BIPOC communities.) Some of the ways these patterns of "colonization" show up in both nonprofit organizations and companies that may or may not meet the criteria for “cults”; but nonetheless are problematic and cause harm within our communities  are:

• Mission-driven vs. Community-centered.  The “mission” and/or the needs/desires of the institution/leaders are put above the needs of the communities they serve (i.e. clients, participants, customers) and employees even in cases where the actions of the organization cause harm to those who interact with it. Healthy organizations intentionally use metrics for evaluation and feedback processes to gather data about their impact and the needs of communities through surveys, focus groups, listening to feedback and grievances within the community, and centering the experiences and needs of the community as being fundamental to their work and mission. They do not put "the mission" above the needs of the community or use it to justify unethical and/or harmful behaviors and impacts.
• Hierarchical and inequitable power structures that reflect along lines of class, race, ability, and gender (often unconsciously) creating and perpetuating longstanding patterns of harm, inequitable access and opportunity, power imbalances, and abuses of power, etc. For this reason, many organizations have started to shift towards collaborative and decentralized models of leadership, and focused education and training in anti-oppression models are essential.
• Lack of effective accountability and grievance processes. Healthy organizations create intentional structures and processes that ensure that leaders, employees, and community members understand standards of conduct and are accountable for their actions and impact. Some examples of this include a well developed and diverse board that has at least 7 members without conflicts of interest (re: nonprofit best practices), clearly outlined grievance and feedback processes for employees and participants, checks and balances within the system, distribution of power, committees and/or employees devoted to handling grievances/complaints and accountability, acknowledging and making amends for harms done,  active engagement in restorative practices and meditation processes, etc.
• Erasure of personal and cultural identities and differences through policies that limit the expression of identities (i.e. sexuality, religious, political, etc.), fear-based compliance and silencing of voices of dissent, lack of inclusion, and lack of power given to those with different backgrounds and perspectives, etc. While a healthy company or organizational culture is important to cultivate and can create strong group cohesion; when branding, policies, uniforms, and other practices seek to exclude or replace existing identities this can lead to unhealthy group dynamics. Healthy organizations value diverse backgrounds, thinking, and approaches; and see the essential and valuable contributions these bring to any organization.
• “Save the world” narratives and marketing of self-aggrandizing narratives that many organizations and companies engage in by over-exagerating the importance, role, or uniqueness of their mission and work. Healthy organizations demonstrate awareness of other organizations who engage in similar types of transformative and social change work in their focus area and/or others who offer similar products or services in the for-profit world - as well as what is unique about their approach, methodology, service, or product. They understand that social change and transformation is a collaborative process as well as the importance of accurately representing and demonstrating the claims they are making; especially when positioning themselves as “best” or better than alternative options.
• Engaging in narrative control through the use of nondisclosure agreements, threats, or other forms of manipulation and coercion. By silencing accounts of harm and unethical conduct within the organization they effectively control the narrative and sharing of information. Withholding or distorting information that would reflect negatively on the organization and/or impairs people’s ability to make fully informed and consent-based decisions through PR, reports, and solicitations that do not accurately describe the activities of an organization or events being reported on; especially to stakeholders, funders, and major donors.
• People are treated as a means to an end, and actions that are unethical or that either intentionally or unintentionally result in harm are dismissed or rationalized through the argument that “the ends justify the means.” Healthy organizations and companies center the experience and needs of community members, participants, employees, clients, and customers; and do not sacrifice the wellbeing and safety of any of these in pursuit of “the mission.”
• Appropriation and consumption of cultural identities (especially BIPOC identities) practices, knowledge, attire, lands, and more without the permission of those who are a part of that cultural identity and without the proper cultural and historical context for the things we are partaking in. Appropriation can also extend to using and/or taking credit for other people's ideas and intellectual property without their permission.

Because of the prevalence of these patterns within organizations and community spaces, I believe it is critical that all organizations (regardless of whether they use a nonprofit or for-profit model) engage in accountability processes, staff training, and community dialogue focused on decolonization, anti-racism, anti-sexism, and other anti-oppression frameworks as well as engage in preventive conversations and measures designed to address systemic barriers, common pitfalls and challenges, and intentionally nourishing and maintaining health organizational practices and community dynamics. It takes a lot of work to examine and de-program the many toxic ideas of leadership and community we've received and build healthy organizations because many of the models we have inherited are harmful.

Please feel free to comment below on other ways you’ve seen patterns and behaviors of “colonization” show up in organizational cultures. By no means is this list of examples comprehensive. How have you seen these patterns show up within the communities and organizations you’ve been a part of? What steps can we take to radically “decolonize” our organizations and communities? What practices and structures do you think best support the development of healthy organizations and community dynamics?

Discuss

### Applied Mathematical Logic For The Practicing Researcher

17 октября, 2021 - 23:28
Published on October 17, 2021 8:28 PM GMT

Asking for a friend[1]: what happened to Richard Hamming's social status after he started asking those pointed questions about the importance of research questions and individual career decisions? Was he, like, actually banished from the lunch table?

Technically, I am modeling for a living

A couple of months ago I've started asking my colleagues during lunch what their definition of a "model" is. This question is important: our job consists of building, evaluating, and comparing models. I am not hoping for an Aristotelean list of necessary & sufficient conditions, but it still appears like a good idea to "survey the land". Also, admittedly, lunch can get a bit boring without challenging questions.

An abstract drawing of a computational model. CGD generated.

I got a range of responses:

"a description of a phenomenon from which you can reason (= a description you can manipulate to tell more about the phenomenon than you would have been able to tell without it)"

"It should be something like a representation of the modelled system without representing it completely. Perhaps most importantly that it preserves the causal relationships between the system elements without completely mapping these elements?"

"an abstraction of reality"

I also ran into this adage again and again (attributed to a different person every time):

"All models are false, but some are useful."

Along similar lines, there is a quote from the influential computational neuroscientist Larry Abbott:

"the term 'realistic' model is a sociological rather than a scientific term."

Alright, survey done, lunch is over. Back to...

In search of tighter concepts

No! I'm not satisfied. What do you mean it's a sociological term? What do you mean they are false? Can a model have a truth value? If a model is a "representation" / "abstraction" / "description" then what exactly is a "representation" / "abstraction" / "description"? This is not some idle philosophical nitpicking, this question is immediately important. As a reviewer, I have to judge whether a model is good (enough). As a researcher, I want to build a good model. I'm not going to devote my career to building models if I don't have a really good idea of what a model is.

I hope you can tell from my extensive use of italicized words that this is a topic I am rather passionate about. If the question of a good model is a sociological question then it's subject to trends and fads[2]. And if the term "model" is broad enough to fit "detailed biophysical models", "abstract phenomenological models", "linear regression" and "a cartoon in Figure 8" under its umbrella, then it's inevitable that our intuitive understanding of what constitutes a good model deviates. Heck, the term is so broad, technically even this should qualify:

An abstract painting of a very attractive albatross that could totally be a fashion model. CGD generated.

So in the spirit of conceptual engineering and dissolving questions, here goes my attempt of laying out what I think of when I think of models. This is obviously not authoritative and it's far from rigorous. This is just my "working definition" which I wrote down to force myself to tighten my terminology.

Mathematical logic to the rescue

Since we mean so many different things by the term "model" it makes sense to start very general, i.e. mathematical. There is indeed a subfield of mathematics called "model theory" that makes some very useful distinctions! I'll trample over all subtleties to get to the core quickly, but consider checking out this or this for accessible introductory reading.

Here goes the central definition:

A model is a (mathematical) object that satisfies all the sentences of a theory.

To make this useful, we have to further define the used terms.

What is a theory? It's a set of sentences. What is a sentence? Well, it's pretty much what you would expect - it's a string of symbols constructed from an alphabet according to some fixed rules. A famous example of a theory is Peano arithmetic, but really the definition is much more general:

1. A dynamical system, given as a set of differential equations[3], is a theory.
2. A cellular automaton, given as a set of transition rules, is a theory.
3. Any recursively enumerable set of sentences of a formal language, given as a set of production rules, is a theory.
An abstract drawing of a cellular automaton. CGD generated.

1. A particular trajectory through state space, f.e. specified through initial conditions.
2. A particular evolution of the cellular automaton, again specified through the initial conditions.
3. A particular Turing machine that implements the production rules, specified through... (you get the idea).

If we are allowed to be even more hand-wavy, then we can also incorporate models à la Tyler Cowen: To "model this [headline]" we have to come up with a theory (a set of sentences) from which the headline follows.

One important thing to note here is that every model "inherits" every property that follows from the theory. But the inverse does not hold[4]: just because a model has a certain property, this property does not necessarily follow from the theory. In general, there will always be multiple models that satisfy a theory, each with different "additional properties" that go beyond what is prescribed by the theory[5].

Defining a model as an object satisfying a theory is broad enough to cover all the ways in which the term is used:

• the entire spectrum of mathematical models, from detailed biophysical to normative Bayesian, is specified by a set of equations (a theory) and instantiated with parameter choices.
• the "cartoon in Figure 8" is one particular (rather toothless) object that satisfies an implicit theory (consisting of a set of conjectured sentences).
• the albatross fashion model... doesn't fit. But you can't have everything, I'm told.

It also includes an interesting pathological case: to model a particular set of observations, we could just come up with a theory that contains all the observations as axioms, but no production rules. Then the observations themselves trivially satisfy the theory. This is clearly useless in some sense[6] (a dataset shouldn't be a model?) - but looking deeper into why it's useless reveals something about what constitutes a good model - or, by extension, a good theory.

Here is my definition:

A good model of a phenomenon is one that allows us to understand something about the phenomenon. If all the models of a theory are good models, the theory is a good theory.

Again, we need to define our terms for this to make sense. What is a phenomenon? A phenomenon is some (conjunction of) physical process(es). It's something out there in the territory. What does understand mean? Understanding a phenomenon means predicting (better than chance level) the state of the phenomenon at time t+1 given the state at time t.

Why does it make sense to set up things like this?

Models with benefits

First, it establishes a neat hierarchy. Understanding is gradual: It goes from non-existing (chance level) to poor (consistently above chance[7]) to great (almost perfect prediction) to complete (100% prediction accuracy).

With this definition, a "black box" deep learning model that is able to predict a percentage of brain activity does provide some understanding about a part of the brain. Similarly, a mean-field model that has "lost" some conversion factor in its units can also still be a good model, as long as it is able to get the direction of the evolution of the state correct.

Second, making predictions the central criterion for model quality helps us avoid unproductive disputes resulting from mismatched terms. The usual example here is "If a tree falls in the forest, does it make a sound?", which can lead to a highly unproductive discussion if asked at the lunch table. But when explanations are evaluated according to their predictive power, misunderstandings are resolved quickly: Either a tape recorder will or won't record airwaves. Either there is or there isn't activation in some auditory cortex.

Third, to have a good theory, you need to demonstrate that all its models are good (according to the definition above). This gets naturally easier if there are fewer models that satisfy the theory, thus incentivizing you to remove as many free parameters from the theory as possible[8]. Ideally, you'll want a unique characterization of a good model from your theory.

Finally, this definition formalizes the "all models are wrong, but some are useful" adage. To get 100% prediction accuracy for a physical process you have to go down to the level of particles. F.e. having a fluid dynamics model of water motion will get you very far in terms of predictive power. In that sense, it's a very good model. But to get even close to 100%, you'll want an atomic model of water. And eventually, if you are pushing for ever more predictive power, you'll have to decompose your problem further and further, and eventually, you will get into very weird territory[9].

Thus, to determine whether a model is good or bad, you have to figure out which phenomenon it is trying to explain and then determine if the model allows you to predict the time-evolution of the phenomenon better than chance level. This is a relatively low bar, but in my experience, it's still not easy to clear. Actually demonstrating that your performance is different from chance requires explicit performance metrics, which are not usually adapted. But that's a different story.

Cliffhanger!

This is almost all I wanted to say on the topic. But I glossed over an important point in that exposition: If a model is a mathematical object, why might we expect that it can predict physical processes out there in the territory? In fact, why should there be any similarity between the solar system and the atom[10]? Why does analogical reasoning work?

I'm glad you ask. Stay tuned - I'll dig into that next time.

[1] Okay, okay, I can't lie to you. That friend is me. I'm worried about getting banished from the lunch table. ↩︎

[2] And it's usually up to an influential "ingroup" to decide what fits in and what doesn't. ↩︎

[3] Plus ZFC, I guess. ↩︎

[4] This inverse only holds when the model uniquely and completely specifies the model, which is pretty hard to achieve in principle. See Logical Pinpointing. ↩︎

[5] One might be tempted to argue that if many different models that all satisfy the same theory, this is evidence that the property actually does follow from the theory. This isn't guaranteed, but it might work in some cases. In Computational Neuroscience, this is the practice of demonstrating that the desired result holds even when the parameter is slightly perturbed. ↩︎

[6] This has some overlap with Chomsky's levels of adequacy: a theory that includes only the observations as axioms has observational adequacy, but neither descriptive nor explanatory adequacy. ↩︎

[7] or below! If you're consistently worse than chance that is very useful information. ↩︎

[8] Thus we arrive at an interesting version of Occam's razor. ↩︎

[9] Let's not talk about quantum stuff on this Substack, okay? ↩︎

[10] Yes, I know that the Bohr model is not the end of the story. But it is still able to explain basically all of chemistry. And also "we don't talk about quantum physics on this Substack". ↩︎

Discuss