Вы здесь

Сборщик RSS-лент

Self-Keeping Secrets

Новости LessWrong.com - 10 ноября, 2019 - 10:59
Published on November 10, 2019 7:59 AM UTC

A magician never reveals his secrets.

The secret behind nearly every magic trick ever performed is available at your local library. Magicial secrets stay secret because they're inconsequential. Unless you are a magician or aspire to become one, you have better things to learn than magic tricks. If magic tricks did anything that mattered they wouldn't be magic tricks. They'd be technology.

Magicians don't need a conspiracy to keep magic tricks secret. It takes work to learn how to do magic. Friction and inertia are sufficient to keep out the riffraff.

This is true of more important subjects too, like computer security. Though zero-day exploits themselves are precious secrets, "how to find" zero-days is public knowledge. And since zero-day exploits have a limited shelf-life, "how to find" zero-days is what matters.

Three may keep a secret, if two of them are dead.

―Benjamin Franklin

Organizations leak like a sponge. Organizations can keep passwords secret most of the time only because a good password is easy to reset. If you're even the slightest bit concerned that your passwords have been stolen then you can re-randomize them. Similarly, an intelligence agency maintains its stockpile of zero-day exploits by constantly replenishing them. To an organization, maintaining secrecy is about restoring secrecy. Techniques can't be kept secret because they change too infrequently to restore secrecy after they get stolen.

In practice, organizations face the opposite problem. Training people is so hard that the limiting factor of an organization's size is how many skilled employees it can hire. The bigger your organization gets the more it'll suffer a regression to the mean. Scaling a large organization is an exercise in dumbing down your employees' jobs to counteract this.

Large organizations can neither keep knowledge secret nor spread it around. In other words, a dependence on knowledge of any kind inhibits the growth of an organization. An organization can scale to the extent it makes its employees'—and especially its customers'—intelligence unnecessary.

SCP-055 is a "self-keeping secret" or "anti-meme".

―internal document, SCP Foundation

Large organizations are precisely those that make knowledge unnecessary. The public school system is, by headcount, among the largest organizations in modern civilization. It must therefore, by necessity, minimize the need for students to learn anything hard[1].

Most adults are employed by large companies. Most adults buy most of our products from large companies. Small businesses are dying out[2]. Modern civilization is increasingly dominated by large organizations. These organizations don't just shape our society. They are our society. We are our jobs. We are the products we use. We are the media we consume. We are our communities.

Our most popular activities are those that scale the best. Those that scale the best are those that require the least thinking, the least skill, the least specialized knowledge, the least individuality. If you want to measure your individuality, ask yourself this: of all the things you do, how much of it is so hard that your friends and coworkers literally can't do it.

  1. By "hard" I mean "conceptual". Schools can effectively force students to learn by rote. As coercive institutions, schools are incapable of forcing students to productively misbehave or otherwise exercise critical thinking ↩︎

  2. Small companies that concentrate a lot of talent in their small number of employees are doing well. But these companies will continue to constitute a small fraction of total employment. ↩︎


Goal-thinking vs desire-thinking

Новости LessWrong.com - 10 ноября, 2019 - 03:31
Published on November 10, 2019 12:31 AM UTC

[Adapted from an old post on my personal blog]

There's a lot of long-running arguments on the internet that basically consist of people arguing past each other due to differing basic assumptions that they don't know how to make explicit, preventing them from noticing the fundamental disagreement. I've noticed a few of these and tried to see if I can make both sides more explicit. In this post I'd like to try to explicate one.

Let's start with a concrete example; there are a number of people who would say that wireheading is a good thing, which is obviously not the general thinking on LW. What's the source of this disagreement? One possible explanation would be to say that the former are saying "happiness is our only terminal value, all other values are subsidiary to it", while the latter say hell no it's not, but I think there's more to it than that.

Without yet saying what I think the fundamental distinction is, let me give another example that I think stems from the same disagreement. Consider this essay -- and this isn't the only thing I've seen along these lines -- which takes the point of view that obviously a rational person would kill themselves, while to me this just seems... dumb.

So what's going on here? What's the actual distinction that leads to such arguments? Again, I can't know, but here's my hypothesis. I think there are two sorts of thinking going on here; I'm going to call them "goal-thinking" and "desire-thinking" (these are my own terms, feel free to devise better ones).

So -- goal thinking is thinking in terms of what I'm calling "goals". Goals are to be accomplished. If you're thinking in terms of goals, what you're afraid of is being thwarted, or having your capacity to act, to effect your goals, reduced -- being somehow disabled or restrained; if your capabilities are reduced, you have less ability to make an effect on the future and steer it towards what you want. (This is important; goal-thinking thinks in terms of preferences about the future.) The ultimate example of this is death -- if you're dead, you can't affect anything anymore. While it's possible in some unusual cases that dying could help accomplish your goals, it's pretty unlikely; most of the time, you're better off remaining alive so that you can continue to affect things. So suicide is almost always unhelpful. Goals, remember, about the world, external to oneself.

Wireheading is similarly disastrous, because it's just another means of rendering oneself inactive. We can generalize "wireheading" of course to anything that causes one to think one has accomplished one's goals when one hasn't. Or of course to having one's goals altered. We all know this argument; this is just the old "murder pill" argument. Indeed, you've likely noticed by this point that I'm just recapitulating Omohundro's basic AI drives.

Another way of putting this is, goals themselves are driving forces.

So what's the alternative, "desire-thinking", that I'm claiming is how many people think? One answer would be to say, this alternative way of thinking is that "it's all about happiness vs unhappiness" or "it's all about pleasure vs pain", thinking in terms of internal experience rather than the external state of the world -- so for instance, people thinking this way tend to focus on unhappiness, pain, and suffering as the general bad thing, rather than having one's capacity to act reduced.

But, as I basically already said above, I actually don't think this gets at the root of the distinction, because there are still things this fails to explain. For instance, I think it fails to explain the suicide article above, or, say, Buddhism; since applying the goal-thinking point of view but applied to internal experiences instead would just lead to hedonism instead. And presumably there are a number of people thinking that way! (Which may include a number of the "wireheading is good" people.) But we can basically group this in as a variant of goal-thinking. How do we explain the truly troublesome cases above, that don't fit into this?

I think what's actually going on with these cases involves not thinking in terms of goals in the above sense at all, but rather what I'm calling "desires" instead. The distinction is that whereas goals are to be accomplished, desires are to be extinguished. From a goal-thinking point of view, you can model this as having one single goal, "extinguish all desires", which is the only driving force; and the desires themselves are, just, like, objects in the model, not themselves driving forces.

So under the desire-thinking point of view, having one's desires altered can be a good thing, if the new ones are easier. If you can just make yourself not care, great. Wireheading is excellent from this point of view, and even killing oneself can work. Indeed, desire-thinking doesn't really think in terms of preferences about the future, so much as just an anticipation of having preferences in the future (about the then-present).

Now while I, and LW more generally, may sympathize more with the former point of view, it's worth noting that in reality nobody uses entirely one or the other. Or at least, it seems pretty clear that even here people won't actually endorse pure goal-thinking for humans (although it's another matter for AIs; this is one of those times when it's worth remembering that LW really has two different functions -- refining the art of human rationality, and refining the art of AI rationality, and that these are not always the same thing). While I don't have a particular link on-hand, this issue has often been discussed here before in terms of preference regarding flavors of ice cream, and how it's not clear that one should resist modifications to this; this can be explained if one imagines that desire-thinking should be applied to such cases.

Thus when Eliezer Yudkowsky says "I wouldn't want to take a pill that would cause me to want to kill people, because then maybe I'd kill people, and I don't want that", we recognize it as an important principle of decision theory; but when someone says "I don't like spinach, and I'm glad I don't, because if I liked it I'd eat it, and I just hate it", we correctly recognize this as a joke. (Despite it being isomorphic.) Still, despite people not actually being all one way or the other, I think it's a useful way of understanding some arguments that have resulted in a lot of people talking past each other.


Neural nets as a model for how humans make and understand visual art

Новости LessWrong.com - 9 ноября, 2019 - 19:53
Published on November 9, 2019 4:53 PM UTC

This is a new paper relating experimental results in deep learning to human psychology and cognitive science. I'm excited to get feedback and comments. I've included some excerpts below.


This paper is about the cognitive science of visual art. Artists create physical artifacts (such as sculptures or paintings) which depict people, objects, and events. These depictions are usually stylized rather than photo-realistic. How is it that humans are able to understand and create stylized representations? Does this ability depend on general cognitive capacities or an evolutionary adaptation for art? What role is played by learning and culture?

Machine Learning can shed light on these questions. It’s possible to train convolutional neural networks (CNNs) to recognize objects without training them on any visual art. If such CNNs can generalize to visual art (by creating and understanding stylized representations), then CNNs provide a model for how humans could understand art without innate adaptations or cultural learning. I argue that Deep Dream and Style Transfer show that CNNs can create a basic form of visual art, and that humans could create art by similar processes. This suggests that artists make art by optimizing for effects on the human object-recognition system. Physical artifacts are optimized to evoke real-world objects for this system (e.g. to evoke people or landscapes) and to serve as superstimuli for this system.

From "Introduction"

In a psychology study in the 1960s, two professors kept their son from seeing any pictures or photos until the age of 19 months. On viewing line-drawings for the first time, the child immediately recognized what was depicted. Yet aside from this study, we have limited data on humans with zero exposure to visual representations.


For the first time in history, there are algorithms [convolutional neural nets] for object recognition that approach human performance across a wide range of datasets. This enables novel computational experiments akin to depriving a child of visual art. It’s possible to train a network to recognize objects (e.g. people, horses, chairs) without giving it any exposure to visual art and then test whether it can understand and create artistic representations.

From "Part 1: Creating art with networks for object recognition"

Figure 2. Outputs from testing whether a conv-net model can generalize to paintings. Results are fairly impressive overall. However, in Picasso painting on the right, the people are classified as "stop sign, frisbee".

Figure 12. Diagram showing how Deep Dream and Style Transfer could be combined. This generates an image that is a superstimulus for the conv net (due to the Deep Dream loss) and has the style (i.e. low-level textures) of the style image. Black arrows show the forward pass of the conv net. Red arrows show the backward pass, which is used to optimize the image in the center by gradient descent.

Figure 13. Diagram showing how the process in Figure 12 can be extended to humans. This is the "Sensory Optimization" model for creating visual art. For humans, the input is a binocular stream of visual perception (represented here as "content video" frames). The goal is to capture the content of this input in a different physical medium (woodcut print) and with a different style. The optimization is not by gradient descent but by a much slower process of blackbox search that draws on human general intelligence.

Figure 14. Semi-abstract images that are classified as “toilet”, “house tick”, and “pornographic” (“NSFW”) by recognition nets. From Tom White’s “Perception Engines” and “Synthetic Abstractions” (with permission from the artist).


Notes on Running Objective

Новости LessWrong.com - 9 ноября, 2019 - 18:40
Published on November 9, 2019 3:40 PM UTC

I've been playing Killer Queen lately at work. It's a ten-person arcade game, five-vs-five. The general idea is that you're a "bumblebear" drone that runs and jumps around trying to win by (a) collecting berries or (b) riding the snail. Some bumblebears will bring berries to "wing gates" and become warriors, who can fly around and kill others. Each side also has one queen, who is like a warrior but also has the ability to dive.

You fly by tapping a button, sometimes very fast (7-15hz depending), and it turns out this hurts my wrists. I have bad wrists, and generally need to be pretty careful. Not being able to play any flying roles, I've been getting a lot of practice playing drone, and wanted to write up some notes.

The two main things drones do are collect berries and ride the snail. Since those are the direct paths to victory, the objectives, we call this "running objective." Typically a team will have one member who always runs objective, and another member who plays "flex", either playing warrior or running objective depending on the needs of the situation.

Presenting everything at an introductory level would take way too long, though, so the rest of this post will be terse and jargony. If you'd like me to explain something ask in the comments, or you can ask your teammates.


  • The longer you hold the button the higher you jump. Making a minimum-sized jump requires only a very slight tap.

  • Your forward speed is the same whether you're walking, jumping, or falling.

  • If you're not making progress because of offensive guards, call for a hive/snail clear.

  • Even though you can't kill anyone, booping others is still very valuable in the right situation. You can boop drones out of gates, boop enemy warriors into your own warriors, and boop warriors out of the hive. Play around with booping to learn which parts are safe and which will kill you.

  • Coordinate with teammates before the round starts to pick a strategy (typically 2-2 berries or 3-1 snail) and figure out if there are any gates you should plan to deny off the bat.

  • Watch the whole board so you have as much warning as possible about what's coming next. Give other players a heads up about important things: number of berries remaining, progress of the snail, warrior counts, gate possession, queen lives. What's worth tracking and calling takes time and depends on your team's preferences, but as dedicated objective you generally have more spare bandwidth than anyone else.

  • Don't get speed unless you literally have nothing better to do. As a drone you're going to die a lot, and the time spent getting speed is not worth it. Speed can help you deliver berries past an inexperienced hive guard, but you want to be building skills that will still work when you play really good players.

  • You can trap standing warriors by jumping onto their heads, but I haven't been able to figure out a good use for this.

Running Berries

  • Go for the hardest spots first, which means the spots that are easiest for the opponents to guard. Top berries on Dusk and Twilight, back berries on Day and Night. Though possibly on Day and Night you should fill front berries first at the very start, if this lets you get an extra cycle in before the other team gets their hive guard up.

  • To get around the hive guard, coordinate with the other objective runner to run the hive at the same time. A solid d-guard can block a solid drone most of the time, but it's very hard to stop two drones at once. On Day you can often boop a hive guard by sticking; see the end of the post.

  • Make sure you and someone else don't go for the same hole. A good convention is you divide left-right based on your position at the cabinet, but make sure you're using the same convention as your teammates. Some teams prefer to call holes.

  • Practice the hard jumps, especially jumping across the top on dusk. Work on delivering the berry into a specific target hole. On Dusk and Twilight learn which holes can be reached by running off the ledge vs jumps of various sizes, and which can be reached by jumping from the bottom ("bottom berries"; lowest three rows on Dusk, lowest two berries on Twilight) vs which need to be filled from the top ("top berries").

  • Figure out the fastest routes on every map.

  • Keep in mind that you're going to crawl back out of a hole, and while you do that the hole is occupied. This means there are times when you should hold back just a little to give another objective runner time to catch up, so you can deliver together. This matters the most for the end of the game, but can also apply to top berries. Don't hold back if it's going to get you killed though.

  • If you're holding a berry and run into another berry you'll kick it. Flying berries that go into holes register immediately. Play around with kicking berries to learn the physics of how your speed affects their speed and angle, because it's not obvious. While warriors absolutely need to learn how to kick so they can clear berries after killing the drone carrying them, berry soccer is only rarely useful for drones and can be a distraction.

Riding Snail

  • When the snail is eating someone, its rider is trapped and is an easy target for warriors. If the snail rider is killed, however, the drone being eaten is immediately free. These combine to mean that you can feed the snail to trap your opponent, get rescued by a teammate, and then ride. If there's high military pressure and you're not sure if rescue is advisable, your o-guard should make the decision.

  • If you're not going to be able to be rescued, and especially if they're not going to be able to be rescued either, it's generally better to just stay on the snail and wait while it slowly eats them.

  • When to hop off the snail and run away vs stay on it and get a few more pixels of progress is a really hard judgement, and mostly depends on whether they're just going to kill you anyway even if you hop off.


Normally, if you jump and hit the ceiling you'll bounce off. But if you hit the ceiling at just the right speed, you'll stick to it for a short time instead. This gets you high enough that you can boop warriors and even the queen, but you have time time it well enough that you're still sticking when you boop them.

Compare bouncing:

To sticking:

Generally you'll want to stick while moving:

Unlike with warriors, tapping while you're sticking has no effect.

To practice sticking, first practice jumping right next to a low ledge, aiming to get your bear's knees even with the underside of the ledge. Once you see what that feels like, try to do the same thing just under a low ledge. Once you're good at sticking you're surprisingly difficult to kill when there's a low ceiling, you can boop hive guards on Day, you can boop warriors out of their own sticks, and you can be one side of a pinch.

With perfect timing it is possible to jump from one ledge and stick on a ledge above that you're not on yet, effectively lipping as a drone, which could be powerful. For example, it should let you boop a hive guard on Night.

Comment via: facebook, r/KillerQueen


Reference Classes for Randomness

Новости LessWrong.com - 9 ноября, 2019 - 17:41
Published on November 9, 2019 2:41 PM UTC

(Follow-up to Randomness vs. Ignorance)

I've claimed that, if you roll a die, your uncertainty about the result of the roll is random, because, in 1/6th of all situations where one has just rolled a die, it will come up a three. Conversely, if you wonder about the existence of a timeless God, whatever uncertainty you have is ignorance. In this post, I make the case that this distinction isn't just a useful analog to probability inside vs. outside a model, but is actually fundamental (if some more ideas are added).

The randomness in the above example doesn't come from some inherent "true randomness" of the die. In fact, this notion of randomness is compatible with determinism. (You could then argue it is not real randomness but just ignorance in disguise, but please just accept the term randomness, whenever I bold it, as a working definition.) This randomness is simply the result of taking all situations which are identical to the current one from your perspective, and observing that, among those, one in six will have the die come up a three. This is a general principle that can be applied to any situation: a fair die, a biased die, delay in traffic, whatever.

The "identical" in the last paragraph needs unpacking. If you roll a die and we consider only the situations that are exactly identical from your perspective, then the die will come up a three either in a lot more or a lot less than 1/6th of them. Regardless of whether the universe is fully deterministic or not, the current state of the die is sure to at least correlate with the chance for a three to end up on top.

However, you are not actually able to distinguish between the situation where you just rolled a die in such a way that it will come up a three, and the situation where you just rolled a die in such a way that it will come up a five, and thus you need to group both situations together. More precisely, you need to group all situations that, to you, look indistinguishable with respect to the result of the die, into one class. Then, if among all situations that belong to this class, the die comes up a three in 1/6th of them, your uncertainty with respect to the die roll is random with probability .mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} 16 for a three. This grouping is based both on computational limitations (you see the die but can't compute how it'll land) and on missing information (you don't see the die). If you were replaced by a superintelligent agent, their reference class would be smaller, but some grouping based on hidden information would remain. Formally, think of an equivalence relation on the set of all brain states.

So at this point, I've based the definition of randomness both on a frequentist principle (counting the number of situations where the die comes up a three vs not a three) and on a more Bayesian-like principle of subjective uncertainty (taking your abilities as a basis for the choice of reference class). Maybe this doesn't yet look like a particularly smart way to do it. But with this post, I am only arguing that this model is consistent: all uncertainty can be viewed as made up of randomness and/or ignorance and no contradictions arise. In the next post, I'll argue that it's also quite useful, in that several controversial problems are answered immediately by adopting this view.


Pricing externalities is not necessarily economically efficient

Новости LessWrong.com - 9 ноября, 2019 - 15:07
Published on November 9, 2019 12:07 PM UTC

[A]s long as externalities exist and are not internalized via Pigouvian taxes, the result is inefficient. The inefficiency is eliminated by charging the polluter an emission fee equal to the damage done by his pollution. In some real world cases it may be difficult to measure the amount of the damage, but, provided that that problem can be solved, using Pigouvian taxes to internalize externalities produces the efficient outcome.

That analysis was accepted by virtually the entire economics profession prior to Coase's work in the field. It is wrong—not in one way but in three. The existence of externalities does not necessarily lead to an inefficient result. Pigouvian taxes, even if they can be correctly calculated, do not in general lead to the efficient result. Third, and most important, the problem is not really externalities at all—it is transaction costs.


English speaking club

События в Кочерге - 9 ноября, 2019 - 15:00
Английский клуб рационалистов - теперь с носителем языка! Разговоры о рациональности и науке, обсуждение когнитивных искажений и техник продуктивности, споры, игры, брейнстормы, мысленные эксперименты. Всё, что мы любим, только на английском.

For the metaphors

Новости LessWrong.com - 9 ноября, 2019 - 02:30
Published on November 8, 2019 11:30 PM UTC

I make use of a lot of analogies, for instance ‘like dancing’ and ‘the ice skating thing’ are particular phenomena I often think about, and I get value from thinking about meta-ethics as if it were romance, or saving the world as if it were a party. I wonder if providing a variety of concrete experiences that other things might be analogized to is a big source of value from doing new things.

For instance, recently I took up knitting and I think there are things about it that my other experiences don’t have. For instance, I got some knitting patterns, and they have this very brief and utilitarian jargon, and a bunch of concepts, and I got a sense of this rich world of actionable and actioned knowledge about how to do a concrete thing, with much doing of it, which is pretty unlike other things I engage in, I am sorry to say. 

I was also struck by the experience of being able to take a relatively simple substance (wool) and turn it into a useful object of the kind one buys in a store (a hat, or it seems like it will be a hat). 

These things are of course what I expect in the abstract, but it is something else to experience things.

I’m not sure how these new experiences compare to the value I have had so far from the activity of knitting, but it seems like much more than the value of a generic hat, and I only have maybe a quarter of one of those.

My current guess is that filling out my repertoire of concrete intuitions about specific kinds of occurrences or relationships between things is pretty helpful.


Catalyst: a collaborative biosecurity summit

Новости LessWrong.com - 9 ноября, 2019 - 00:37
Published on November 8, 2019 9:37 PM UTC

Applications for Catalyst, a collaborative biosecurity summit, are now open.

On February 22, 2020, at The Laundry in San Francisco, a diverse group of synthetic biologists, policymakers, academics, and biohackers will come together to discuss one of the biggest emerging questions of the 21st century—how do we engineer a future enhanced by biotechnology and not endangered by it?

If you’d like to meet other individuals invested in the future of biotechnology, participate in forward-thinking conversations, or brainstorm solutions to open problems in biosecurity, apply by December 7th to secure a spot.


Рациональное додзё. Систематизация

События в Кочерге - 8 ноября, 2019 - 19:30
Систематизация – подход к решению проблем через изменение среды, в которой мы находимся. Мы поговорим о принципах дизайна и потренируемся находить внешние решения внутренним проблемам.

Рациональное додзё. Систематизация

События в Кочерге - 8 ноября, 2019 - 19:30
Систематизация – подход к решению проблем через изменение среды, в которой мы находимся. Мы поговорим о принципах дизайна и потренируемся находить внешние решения внутренним проблемам.

How to Get a Duplicate Receipt

Новости LessWrong.com - 8 ноября, 2019 - 17:30
Published on November 8, 2019 2:30 PM UTC

Let's say you have a credit card charge from a store, and you need a receipt. Maybe you aren't sure whether you made it, and want to see what's what purchased. Or you need to file for reimbursement but you lost the receipt. What can you do?

You might think you could walk into the store, give them your credit card, and they could bring up a list of your past transactions. But even though the industry has moved to automated point-of-sale machines, most retailers don't have good systems for looking up transactions.

Registers each keep a transaction log, in chronological order. If you come in with just what's on the credit card statement (day and amount) they're going to need to do an awkward amount of work to find the transaction. Specifically, they'll need to review that day's logs, register by register, manually scanning for a transaction of the right amount. Once they find it they can check that the credit card number matches, and print off a duplicate.

If you know what register you were at, and what time you made the purchase, it's much less work for them. It's very surprising to me that their computer systems don't support searching, but asking at three different stores it sounds like they generally don't. I'm not sure why?

Comment via: facebook


Levers error

Новости LessWrong.com - 8 ноября, 2019 - 11:35
Published on November 8, 2019 8:35 AM UTC

Anna writes about bucket errors. To gloss the idea: sometimes two facts are mentally tracked by only one variable; in that case, correctly updating the belief about one fact can also incorrectly update the belief about the other fact, so it is sometimes epistemic to flinch away from the truth of the first fact (until you can create more variables to track the facts separately).

I think there's a sort of conjugate error: two actions are bound together in one "lever". An action is a class of motor outputs, and a lever is a thing actually available to the mind to decide to do or not.

For example, I want to clean my room. But somehow it feels pointless / tiring, even before I've started. If I just started cleaning anyway, I'd get bogged down in some corner, trying to make a bunch of decisions about where exactly to put lots of futzy random objects, tiring myself out and leaving my room still annoyingly cluttered. It's not that there's a necessary connection between cleaning my room and futzing around inefficiently; but the only lever I have right now that activates the "clean room" action also activates the "futz interminably" action.

What I want instead is to create a lever that activates "clean room" but not "futz". When I do that, I feel motivated to clean and do so efficiently. (I do this by picking a few specific spots, e.g. "this desk" and "that pile of stuff in the way", picturing what clean and neat would look like, and then holding the intention towards that state.)

I find many examples of this sort of levers error in my life.

  • For example, it's taken me a long time to start writing things to share with other people; for a long time, whenever I'd start to write up an idea, I'd somehow end up "desecrating" the idea, twisting it into some kind of high-school-essay monstrosity, because that was the only way I knew how to write paragraphs. I felt anti-motivated to actually write stuff, even though "writing up some ideas" seemed like a pretty good action to take.
  • I find lots of examples in my ways of interacting with people. For example, for a long time I didn't know how to be nice / receptive / spacious with a person, without also being deferential / self-effacing. Whenever I'd start to be receptive to what the other person was feeling / thinking / wanting, I'd also start ignoring / overriding my thoughts and desires.
  • Another example: sometimes I want to maintain a connection with another person, and I believe that to do that I have to hear and understand about their feelings; but the only way I know how to hear about their feelings also involves dissociating / holding them at a distance, and that can be damaging to the connection in a different way.

The general pattern is something like: I want to do X to acheive some goal, but the only way (that I know how right now) to do X is if I also do Y, and doing Y in this situation would be bad. I only have the one lever, even though there's two kinds of actions that might be doable independently of each other. I can look for ways to do X without also doing Y, and I can look for information I can glean and then use to indicate whether or not it would be good to also do Y in this situation. (Maybe sometimes it's good to futz around with stuff in the corner of my room, e.g. if I'm trying to pack up for a move and want to get rid of a bunch of stuff.)

To complete the conjugation from Anna's post: flinching away from action toward a goal is often about protecting your goals.


small worlds

Новости LessWrong.com - 8 ноября, 2019 - 10:13
Published on November 8, 2019 7:13 AM UTC

(consider skipping 0, 1, 2, and 3)

0. a mind thinks using an ontology, which answers "what things can exist, how do they relate, and how do they change?".
00. an atemporal ontology is a collection of types of things that can exist, along with ways in which those things can relate to each other.
01. a situation sustained by an ontology is a collection of some things, i.e. instances of the types from the ontology, along with instances of relations between those things.
02. a temporal ontology is an atemporal ontology O along with ways in which situations in O can develop into other situations in O as time passes.
03. for temporal ontology, write simply "ontology".
04. an ontology O may contain things that emerge as regularities in a mind that thinks using O.

1. a goal is referenced using an axiology, which answers "what do i care about?".
10. an axiology may be an ontology.
11. an axiology may be something other than an ontology. for example, an axiology may refer to a mind being in a certain state, which is a different sort of reference than the reference executed by an ontology.
110. example: a certain part P of a human H wants to predict with certainty that they will always have access to affection. that is: H has "access to affection" in their ontology; H's mind predicts whether the situation will always contain things bearing the relation "(H has) access to affection"; and the part P causes H to act until that prediction is certain. the prediction is not (necessarily) a thing in an ontology used by H, but H still operates as though they have a goal about the state of the prediction.
111. example: conscious experience; human connection; pleasure; thought. (a human may separately have these things in their ontology.)
12. an axiology may be continuous and overlapping with a mind's ontology, by valuing (terminally or instrumentally) things or situations in the ontology.
13. an axiology A may extend outside of a mind's ontology O. for example, A may be another ontology that strictly contains O, or A may be something other than an ontology.

2. a change is referenced using a skillset, which answers "what can be done?".
20. a skillset is a collection of skills.
21. a skill is a way of doing something. doing something means to change the situation in specific ways.
210. for example: picking up a hammer; buying stock; painting a picture; sorting a list; firing employees; hiking; updating beliefs; lifting weights; making war; programming a computer; eating; becoming sleepy; rebeling; dissociating; flirting; drumming; passing laws; speaking; reading; sneezing; grinding a lens.
211. a skill may be cartesian, e.g. opening a door; reflective cartesian, e.g. drinking coffee, amending the rights of Congress, or dissociating; or naturalized, e.g. paying attention, Focusing, or following the Golden Rule.
212. acting randomly is a skill if it is weilded as a subskill by another skill, such as (inefficient but higher exploration-value) searching or evading adversarial prediction, but acting randomly is not a skill on its own.

3. a layout in an ontology O is a collection of paths between situations in O.
30. a skill generates paths from situations where it is applied, to the resulting situations.
31. a skillset generates a layout in an ontology comprising all the paths generated by the skills in the skillset.
32. a situation considered in a layout has a neighborhood in that layout.

4. a small world is a situation and a layout in the neighborhood of that situation.

5. an actor lives in a small world.
50. an actor A is a connection between an axiology Ax and a skillset S via a mind with an ontology O.
51. every actor is situated: it exists inside of a situation in O.
52. the small world of an actor A is A's situation, along with the layout near that situation generated by the skillset of A, in the union ontology of A's ontology and axiology.
53. every actor A lives in its small world: A uses S to navigate the layout generated by S around A's situation in O in pursuit of A's goals in Ax.
54. skillsets are hence reflective/creative: in skillfully navigating the local neighborhood around its situation, an actor amplifies and creates skills, which changes the layout, and hence changes the task of the actor.
55. in "plain english": there are things that an actor can perceive (notice) and conceive (mentally manipulate) to greater and lesser extents; these extents create a landscape of possibilities and impossibilities for the actor to move in and thence affect the world.

6. consciousness is for mediating between small worlds.

7. all actors, in the limit and also by default, are at rest.
70. an at-rest actor's situation, comprehensively viewed, is such that they are already at their goal state.
701. for example, the goal "i will always have food" may be satisfied in a small world that contains an eternal picture of the existing food supply chain.
71. hence a lone small world tends to be sterile, ethereal, contentless, trivial: everything is stationary, necessary, complete, and meaningless.
710. a small world is not necessarily sterile: there are possible small worlds that are also large worlds.

8. when each actor in a population of actors is actually at rest, the population becomes the substrate in which new kinds of actors (i.e. actors with novel small worlds) arise and move.

-society is a system of actor-substrate relations. some are reciprocal, some are hierarchical.
-small worlds can be sustained within a subset of a human mind, or across multiple human minds.
-incentive to cause actors to be at rest: like code used by other code, or biological subsystems used by other subsystems, actors that use other actors as a substrate are incentivize to incentivize those other actors to be at rest.
-small worlds can overlap, be disjoint, break, collide, connect, form a patchwork, generate other small worlds, grow, shrink, mutate, form ecologies, be copied, be viral, go to war (vergonha, the holocaust), be open or closed, recruit substrate, mimic other small worlds, and so on.
-all small worlds intersect in reality.
-all small worlds are impoverished with respect to reality.
-some small worlds try to encompass all other small worlds. some small worlds try to encompass reality.
-horror is when a big reality rears its head outside your small world.


A First Sketch of the Nature of Heat (Novum Organum Book 2: 15-25)

Новости LessWrong.com - 8 ноября, 2019 - 06:42
Published on November 8, 2019 3:42 AM UTC

This is the eleventh post in the Novum Organum sequence. For context, see the sequence introduction. For the reading guide, see earlier posts in the sequence.

We have used Francis Bacon's Novum Organum in the version presented atwww.earlymoderntexts.com. Translated by and copyright to Jonathan Bennett. Prepared for LessWrong by Ruby.

[[In the previous section, Bacon introduced his "three tables": his careful collection of data and observations that are core to building up his scientific method.

These tables are:

1) A table of presence which lists many examples where phenomena of interest in presence, e.g. many examples of things where we have heat.

2) A table of nearby essence. His example is heat and to discriminate its true heat, Bacon looks for examples of things that resemble those in the table of presence yet are lacking the heat. For the example, the light of the moon (cold) is contrasted with the light of the sun (hot) which is interesting given they are both heavenly bodies.

3) A table of degrees or comparison where are examples are brought where the amount of perceived heat differs in degree between things. This is also useful in discriminating the true underlying cause and and nature of heat.]]

Aphorism Concerning the Interpretation of Nature: Book 2: 15-25

15. The job of these three tables is—in the terminology I have chosen—to present instances to the intellect. After the presentation has been made, induction itself must get to work. After looking at each and every instance we have to find a nature which

  • is always present when the given nature (in our present case: heat) is present,
  • is always absent when the given nature is absent,
  • always increases or decreases with the given nature, and
  • is a special case of a more general nature

(I mentioned this last requirement in 4). If the mind tries to do this •affirmatively from the outset (which it always does when left to itself), the result will be fancies and guesses and ill-defined notions and axioms that have to be adjusted daily. (Unless like the schoolmen* we choose to fight in defence of error; and in that case how well an axiom fares will depend ·not on how much truth it contains but· on the ability and strength of its defender.) It is for God (who designed and gave the forms), and perhaps also for angels and higher intelligences, to have an immediate •affirmative knowledge of forms straight away. This is certainly more than man can do. We have to proceed at first through

[[*Schoolmen: Aristotelian scholars.]]


[Bacon will now be likening scientific procedure to a kind of chemical analysis, in which various components of a complex liquid are distilled off by heat, leaving the residue in which we are interested.]

So we have to subject the nature ·in which we are interested· to a complete dismantling and analysis, not by fire but by the mind, which is a kind of divine fire. The first task of true induction (as regards the discovery of forms) is to reject or exclude natures that

  • are not found in some instance where the given nature is present, or
  • are found in some instance from which the given nature is absent, or
  • are found to increase in some instance when the given nature decreases, or
  • are found to decrease when the given nature increases.

After these rejections and exclusions have been properly made, and all volatile opinions have been boiled off as vapour, there will remain at the bottom of the flask (so to speak) an affirmative form that is solid, true and well defined. It doesn’t take long to say this, but the process of doing it is lengthy and complex. Perhaps I’ll manage not to overlook anything that can help in the task.

17. I have to warn you—and I can’t say this too often!—that

When you see me giving so much importance to forms, do not think I am talking about the ‘forms’ that you have been used to thinking about.

·Treating my forms as your ‘forms’ in the present context would be wrong in two ways·. (1) I’m not talking here about composite forms, the ones in which various simple natures are brought together in the way the universe brings them together—the likes of the forms of lion, eagle, rose, gold, and so on. It will be time to treat of these when we come to hidden processes and hidden microstructures, and the discovery of them in so-called substances or composite natures.

(2) In speaking of ·forms or· simple natures, I’m not talking about abstract forms and ideas which show up unclearly in matter if indeed they show up in it at all. When I speak of ‘forms’ I mean simply the objective real-world laws of pure action* that govern and constitute any simple nature—e.g. heat, light, weight—in every kind of matter and in anything else that is susceptible to them. Thus the ‘form of heat’ or the ‘form of light’ is the same thing as the law of heat or the law of light; and I shan’t ever use abstractions through which I step back from things themselves and their operations.

[*Bacon doesn’t explain actus purus. In each of its other three occurrences he connects it with laws, and his meaning seems to be something like: ‘the laws governing the pure actions of individual things, i.e. the things they do because of their own natures independently of interference from anything else’. If x does A partly because of influence from something else y, then x is not purely •active in respect of A because y’s influence gives A a certain degree of •passivity. From here on, actus purus will be translated by ‘pure action’.]

[In the next sentence, ‘rarity’ is cognate with ‘rare’ in the sense of ‘thin, attenuated, not dense’.] So when I say (for instance) in the investigation of the form of heat

  • ‘reject rarity ·from the list of simple natures that constitute heat·’, or
  • ‘rarity does not belong to the form of heat’,

·I may seem to be talking about an abstract property rarity, but what I am saying can just as well be said without any noun purporting to refer to any such abstraction. For· those statements are tantamount to

  • ‘It is possible for us to make a dense body hot’, or
  • ‘It is possible for us keep or remove heat from a rare body',

·where ‘rarity’ and ‘denseness’ give way to ‘rare’ and ‘dense’·.

You may think that my forms also are somewhat abstract, as they mix and combine things that are very different from one another. ·This complaint might come from your noticing that·

  • the heat of heavenly bodies seems to be very unlike the heat of fire,
  • the relatively durable redness of a rose (say) is very unlike the ·transient shimmering· redness that appears in a rainbow, an opal, or a diamond, and the different kinds of death—by drowning, burning, stabbing, stroke, starvation—are very unalike;

yet they share the nature of heat, redness and death respectively. If you do have that thought, this shows that your mind is captive to •habit, to •things taken as a whole ·and not subject to analysis or bit-by-bit examination·, and to •men’s opinions. For it is quite certain that these things, however unalike they may be, agree in the form or law that governs heat, redness and death (respectively); and human power can’t possibly be freed from the common course of nature, and expanded and raised to new powers and new ways of operating, except by discovering of forms of this kind. This •union of nature is the most important thing I have to talk about; but when I have finished with it I shall take up, in the proper place, the •divisions and veins of nature, both the ordinary ·superficial· ones and also the ones that are more internal and true. ·By the ‘union of nature’ I mean the coming together of disparate things under a single form. By the ‘division and veins of nature’ I mean the complexities in which disparate structures and functions come together in a single thing·.

18. I should now provide an example of the exclusion or rejection of natures that are shown by the Tables of Presentation not to belong to the form of heat. All that is needed for the rejection of any nature ·from the form we are investigating· is a single ·contrary· instance from one of the tables; for what I have said makes it obvious that any conjecture ·of the type ‘Nature N belongs to form F’· is knocked out by a single contrary instance. But I shall sometimes cite two or three such instances—for clarity’s sake and to provide practice in using the tables.

An example of exclusion or rejection of natures from the form of heat:

(1) reject: elemental nature because of the rays of the sun

(2) reject: heavenly nature because of ordinary fire, and especially underground fires, which are the most completely cut off from the rays of heavenly bodies

(3) reject: how fine-grained a body’s structure is because of the fact that all kinds of bodies (minerals, vegetables, skin of animals, water, oil, air, and so on) become warm simply by being close to a fire or other hot body

(4) reject: being attached to or mixed with another body that is hot because of the fact that red-hot iron and other metals give heat to other bodies without losing any of their own weight or substance

(5) reject: light and brightness because of boiling water and ·hot· air, and also metals and other solids that become hot but not enough to burn or glow

(6) reject: light and brightness because of the rays of the moon and other heavenly bodies (except the sun)

(7) reject: light and brightness because of the fact that red-hot iron has more heat and less brightness than the flame of alcohol

(8) reject: rarity because of very hot gold and other metals that have the greatest density

(9) reject: rarity because of air, which remains rare however cold it becomes

(10) reject: change in a body’s size or shape because of red-hot iron, which doesn’t become larger or change its shape

(11) reject: change in a body’s size or shape because of the fact that in thermometers, and the like, air expands without becoming noticeably warmer

(12) reject: destructive nature, or the forceful addition of any new nature because of the ease with which all bodies are heated without any destruction or noticeable alteration

(13) reject: expanding or contracting motion of the body as a whole because of the agreement and conformity of similar effects displayed by both heat and cold

(14) reject: the basic natures of things (as distinct from properties they have through antecedent causes) because of the creation of heat by rubbing things together There are other natures beside these; I’m not offering complete tables, but merely examples.

Not a single one of the ‘reject:’ natures belongs to the form of heat. In all our dealings with heat we can set those aside.

19. The process of exclusion is the foundation of true induction; but the induction isn’t completed until it arrives at something affirmative. Of course the excluding part ·of our work· is itself nothing like complete, and it can’t be so at the beginning. For exclusion is, obviously, the rejection of simple natures; so how can we do it accurately when we still don’t have sound and true notions of simple natures? Some of the notions that I have mentioned (such as the notions of elemental nature, heavenly nature and rarity) are vague and ill defined. I’m well aware of, and keep in mind, how great a work I am engaged in (namely making the human intellect a match for things and for nature); so I am not satisfied with what I have said up to here. I now go further, and devise and supply more powerful aids for the intellect—aids that I shall now present. In the interpretation of nature the mind should be thoroughly prepared and shaped up, so that it will at each stage settle for the degree of certainty that is appropriate there, while remembering (especially at the beginning) that the answer to ‘What is this that we have before us?’ depends to a great extent on what will come of it later on.

20. Truth emerges more quickly from error than from confusion, ·which implies that it can be worthwhile to aim for clarity even at the risk of going wrong·. So I think it will be useful, after making and weighing up three tables of first presentation (such as I have exhibited), to give the intellect permission to try for an interpretation of nature of the affirmative kind on the strength of the instances given in the tables and also of any others that may turn up elsewhere. I call this kind of attempt •‘permission for the intellect’ or •‘sketch of an interpretation’ or—·the label I shall actually use in this work·—•the ‘first harvest’.

A first harvest of the form of heat

Something that is perfectly clear from what I have said earlier should be borne in mind here, namely that the •form of a thing is present in each and every instance of the thing; otherwise it wouldn’t be its •form; from which it follows that there can’t be any counter-instances ·where the thing is present and the form isn’t·. Still, the form is much more conspicuous and obvious in •some instances than in others, namely in •those where the nature of the form is less restrained and obstructed and limited by other natures. Instances of •this kind I call ‘luminous’ or (·most of the time·) ‘revealing’ instances. So now let us proceed to the first harvest concerning the form of heat.

In each and every case of heat the cause of the nature of which heat is a special case appears to be motion. This shows most conspicuously in flames, which are on the move all the time, and in boiling or simmering liquids, which are also constantly in motion. It is also shown when motion stirs heat up or increases it—as happens with bellows and with wind (Third Table 29) and with other kinds of motion (28 and 31). It is also shown when fire and heat are extinguished by any strong compression, which checks and stops the motion (see 30 and 32). It is shown also by the fact that all bodies are destroyed or at any rate significantly changed by any fire or strong heat, which makes it quite clear that heat causes a tumult and agitation and lively motion in the internal parts of a body, which gradually moves it towards dissolution.

In certain cases heat generates motion and in certain cases motion generates heat, but that isn’t what I am saying when I say that motion is like a genus in relation to heat ·as one of its species·. What I mean is that heat itself is nothing but motion of a certain specific kind; I’ll tell you soon what special features of a case of motion make it qualify as a case of heat. Before coming to that, though I shall present three cautions that may be needed to avoid unclarity about some of the terms I shall be using.

·First caution: My topic is heat, not heat-as-we-feel-it·. Heat as we feel it is a relative thing—relative to humans, not to the world; and it is rightly regarded as merely the effect of heat on the animal spirits. Moreover, in itself it is variable, since a single body induces a perception of cold as well as of heat, depending on the condition of the senses. This is clear from the item 41 in the Third Table [here].

·Second caution: My topic is heat, not the passing on of heat·. Don’t confuse the form of heat with the passing on of heat from body to body, for heat is not the same as heating. Heat is produced by the motion of rubbing something that at first has no heat; and that’s enough to show that the transmission of heat is no part of the form of heat. And even when something is heated by another hot thing’s coming close to it, that doesn’t come from the form of heat; rather, it depends entirely on a higher and more general nature, namely the nature of assimilation or self-multiplication, a subject that needs to be investigated separately. [See here.]

·Third caution: My topic is heat, not fire·. Our notion of fire is a layman’s one, and is useless ·for scientific purposes·. What it counts as ‘fire’ is the combination of heat and brightness in a body, as in ordinary flame and bodies that are red hot. [Red-heat is treated as a kind of ‘burning’ in item 24 here.]

Having guarded against verbal misunderstandings, I now at last come to the true specific differences which qualify a case of •motion (·genus·) to count as a case of •heat (·species·).

The first difference then is this. Heat is an expansive motion in which a body tries expand to a greater size than it had before. We see this most clearly in flame, where the smoke or thick vapour obviously expands into flame.

It also appears in any boiling liquid, which can be seen to swell, rise and bubble, and goes on expanding itself until it turns into a body that is far bigger than the liquid itself, namely into steam, smoke, or air.

It appears also in all wood and ·other· flammable things, where there is sometimes sweating and always evaporation.

It is shown also in the melting of metals. Because they are highly compact, metals don’t easily expand and dilate; but their spirit expands, and wants to expand further; so it forces and agitates the lumpier parts into a liquid state. If the metal becomes hotter still, it dissolves and turns much of itself into a volatile substance.

It appears also in iron or rocks: they don’t liquefy or run together, but they become soft. Similarly with wooden sticks, which become flexible when slightly heated in hot ashes.

But this kind of motion is best seen in air, which a little heat causes to expand—see Third Table 38 [here].

It shows up also in the contrary nature, namely cold. For cold contracts all bodies—makes them shrink—so that in a hard frost nails fall out of walls, bronze vessels crack, and heated glass when exposed to cold cracks and breaks. Similarly, a little cooling makes air contract, as in 38. But I’ll say more about this when I deal properly with cold.

It’s no wonder that heat and cold should exhibit many actions in common (for which see the Second Table 32). This first specific difference ·helping to denarcate the species heat within the genus motion· concerns a feature of heat that is diametrically opposite to a feature of cold, because whereas heat expands cold contracts; but the third and fourth differences (still to come) belong to the natures both of heat and of cold.

The second difference is a special case of the first, namely: Heat is a motion in which the hot body •expands while it •rises. This is a case of mixed motion, of which there are many—e.g. an arrow or javelin •rotates while it •flies forward. Similarly the motion of heat is an expansion as well as a movement upwards.

This difference appears when you put a poker into a fire. If you put it in upright and hold it by the top, it soon burns your hand; if you put it in at the side or from below, it takes longer to burn your hand.

It can also be seen in fractional distillation, which men use for ·extracting essences from· delicate flowers that soon lose their scent. It has been found in practice that one should place the fire not below ·the distilling retort· but above it, so as to burn less. For all heat, not only flame, tends upward.

This should be tried out on the opposite nature, cold, to learn whether cold contracts a body downward as heat expands it upward. Here’s how to do it. Take two iron rods or glass tubes of exactly the same dimensions, warm them a little and place a sponge steeped in cold water or snow at the bottom of the one, and a similar one at the top of the other. I think that the end of the rod that has snow at the top will cool sooner than the end of the rod with snow at the bottom—the opposite of what happens with heat.

The third specific difference is this: heat is a motion that isn’t expansive uniformly through the whole ·hot· body, but only through its smaller particles; and this expansion ·in any one particle· is at the same time checked, repelled, and beaten back ·by the expansions of other particles·, so that there’s a back-and-forth motion within the body, which is irritated by all the quivering, straining and struggling that goes on; and from that comes the fury of fire and heat.

This ·specific· difference is most apparent in flames and in boiling liquids, where there are continual little rises and falls across their surface.

It also shows up in bodies that are so compact that when heated or ignited they don’t swell or expand in bulk—e.g. in red-hot iron, in which the heat is very sharp.

And it is apparent in hearth fires, which burn brightest in the coldest weather.

It also shows in the fact that when the air in a calendar glass [see item 38 here] expands without obstacles or counter-pressures, and thus expands at the same rate throughout, there is no perceptible heat. Also when an enclosed body of ·compressed· air escapes, no great heat is observed; that is because although the air bursts out with the greatest force, its only expansive motion is a motion of the whole, with no back-and-forth motions in the particles. . . .

It is also shown in this, that all burning acts on minute pores in the body in question, so that burning digs into the body, penetrating and pricking and stinging it like the points of countless needles. . . .

And this third specific difference is shared with the nature of cold. For in cold the contractive motion is checked by a tendency to expand, just as in heat the expansive motion is checked by a tendency to contract. Thus, whether the particles of a body work inward or outward, the mode of action is the same though the degree of strength may be very different; because on the surface of the earth we don’t have anything that is intensely cold. [See item (3) here.]

The fourth specific difference is a special case of the third. It is that the motion of pricking and penetrating must be fairly fast, not sluggish, and must go by particles—very small ones but a bit bigger than the smallest.

This difference is apparent when you compare the effects of •fire with the effects of •time or age. Age or time makes things wither, consumes and undermines them, reduces them to ashes, just as much as fire does, though it acts on even smaller particles than fire acts on; because that motion is very slow and acts on very tiny particles, there is no detectable heat.

It is also shown by comparing the dissolution ·in acids· of iron and gold. Gold is dissolved without any heat being stirred up, whereas iron, when it is dissolved about as quickly as gold, starts up a violent heat. This is because the solvent for gold enters the gold gently and works at a level of very small particles, so that the particles of the gold give way easily; whereas the solvent for iron enters the iron roughly and forcibly, and the particles of the iron are more stubborn.

It is also apparent in some gangrenes and cases of rotting flesh, which don’t arouse much heat or pain because the rotting process operates at the level of such tiny particles.

I offer this as the •first harvest—or •sketch of an interpretation—concerning the form of heat, made by way of •permission to the intellect [these three labels are introduced in 20 here.].

The form or true definition of heat can be derived from this first harvest. (I’m talking about heat considered absolutely, not heat relative to the senses.) Here it is, briefly:

•Heat is an expansive motion that is resisted, and that fights its way through the smaller particles ·of the hot body·.

Special case of this expansion:

•While expanding in all directions ·the hot body· has a tendency to rise.

Special case of the struggle through the particles:

•It is not very slow; rather it is fast and has some force.

This tells us how in practice to create heat. Here is the story:

In some natural body, arouse a motion to expand; and repress this motion and turn it back on itself so that the expansion doesn’t proceed evenly, but partly succeeds and is partly held back.

If you do that you will undoubtedly generate heat. It makes no difference whether

•the body is made of earthly elements or contains heavenly substances,
•is luminous or opaque,
•is rare or dense,
•is spatially expanded or still of its original size,
•tends towards dissolution or keeps its original condition,
•is animal, vegetable, or mineral (water, oil or air),

or any other substance that is capable of the motion described. Sensible heat is the same, but considered with reference to the senses. Let us now proceed to further aids.

[That last remark refers to the ‘aids’ that were promised in 19 here; the first such ‘aid’ has been 20. A reminder about ‘the tables of first presentation’:

•the first table, of essence and presence, starts here;
•the second table, of divergence or nearby absence, starts here;
•the third table, of degrees or of comparison, starts here;
•‘the table of exclusion or rejection’ starts here;
•‘the first harvest’ starts here.

This reminder may be useful as a guide to Bacon’s next remark.]

21. So much for the tables of •first presentation and of •rejection or exclusion, and the •first harvest based on them. Now we have to proceed to the other aids to the intellect in the interpretation of nature and in true and perfect induction. I’ll present them in terms of heat and cold whenever tables are appropriate; but when only a few examples are needed I’ll take them from all over the place, so as to give my doctrine as much scope as possible without creating confusion.

[We are about to meet the phrase ‘privileged instances’. The Latin praerogativa instantarum strictly means ‘privilege of instances’, but Bacon always handles it as though it stood for a kind of instance, not a kind of privilege. The use of ‘privilege’ to translate praerogativa is due to Silverthorne, who relates it to the centuria praerogativa in ancient republican Rome—the aristocrats’ privilege of voting first and thus having the best chance to influence the votes of others.]

My topics will be, in this order:

1. privileged instances

2. supports for induction

3. the correcting of induction

4. adapting the investigation to the nature of the subject

5. which natures should be investigated first, and which later

6. the limits of investigation, or a synopsis of all natures in the universe

7. practical consequences

8. preparations for investigation

9. the ascending and descending scale of axioms.

[There are twenty-seven classes of privileged instances, some with a number of sub-classes. Bacon’s discussion of them runs to the end of the work. The other eight topics were to have been dealt with in later instalments of the Great Fresh Start, which he never wrote.]

22. Class 1 of privileged instances: solitary instances. Those are ones in which the nature we are investigating

appears in things that have nothing else in common with other things that have that nature,

or ones in which the nature we are investigating

does not appear in things that have everything else in common with other things that do have that nature.

·I put these first · because it is clear that they save us from detours, leading quickly and securely to exclusions, so that a few solitary instances are as good as many.

Suppose for example that we are investigating the nature of colour: in that context prisms, crystals, dew-drops and the like, which make colours in themselves and project them outside themselves onto a wall, are solitary instances. For they have nothing else in common with the colours inherent in flowers, coloured stones, metals, woods, etc.—i.e. nothing but colour. From which we can easily draw the conclusion that colour is merely a modification of the light that the object takes in. With prisms, crystals etc. the light is modified by the different angles at which the light strikes the body; with flowers, coloured stones etc. it is modified by various textures and microstructures of the body. These instances are •resemblance-solitary.

In that same investigation of light: the distinct veins of white and black in a piece or marble, and the variegation of colour in flowers of the same species, are solitary instances. The black and white streaks in marble have almost everything in common except their colour, and so do the streaks of pink and white in a carnation. From this we can easily infer that colour doesn’t have much to do with the intrinsic nature—·the microscopic fine texture·—of a body, but only on the quasi-mechanical arrangement of its larger parts. These instances are •difference-solitary. . . .

23. Class 2 of privileged instances: shifting instances. Those are ones where the nature under study is •shifting towards being produced when it didn’t previously exist, or •shifting towards non-existence when it existed before. Shifting instances, whichever kind of shift they involve, are always twofold, or rather it is one instance in which the movement is continued until it reaches the opposite state.

[At this point some material is removed, and will be reinserted as a paragraph between *asterisks* below; it is easier to understand there than it would be here.]

Here is an example of a shifting instance. Suppose we are investigating whiteness: shifting instances in which the shift is towards production or existence ·of whiteness· are

unbroken glass shifting to powdered glass ordinary water shifting to water shaken up to make foam.

Plain glass and water are transparent, not white, whereas pounded glass and foaming water are white, not transparent. So we have to ask what happened to the glass or water in this shift. Obviously, the form of whiteness is brought in by the pounding of the glass and the shaking of the water; but we find that nothing has occurred except the breaking up of the glass and water into small parts, and the introduction of air. So we have this result:

Two bodies, air and water (or: air and glass) which are more or less transparent come to exhibit whiteness as soon as they are broken up into small bits ·and the bits are mixed·, this whiteness being brought about by the unequal refraction of the rays of light.

This is a big step towards discovering the form of whiteness.

*Such instances don’t just lead quickly and securely to exclusions, but also narrow down the search for the affirmation or the form itself [‘exclusion’ and ‘affirmation’ are introduced in 15 here]. For the form of a thing must be something that is introduced by a shift, or removed and wiped out by a shift in the other direction. Of course every exclusion supports some affirmation, but the support is more direct when the exclusion comes from one case rather than from a number of cases. And my discussion has made it clear that the form that comes to light in a single instance leads the way to the discovery of it in all the rest. And the simpler the shift, the more value we should attach to the instance. And another thing: shifting instances are of great value in the practical part ·of scientific inquiry·: a shifting instance exhibits •the form ·under investigation· linked with •the cause of its existing (or the cause of its not existing); that provides great clarity in one instance and an easy transition to others. But shifting instances create a certain danger against which I should warn you: they may lead us to link the form too closely to its efficient cause, and so encourage a false view of the form, drawn from a view of the efficient cause. The efficient cause is always understood to be merely the vehicle for or bearer of the form. It is not hard to avoid this danger in a properly conducted exclusion.*

I should give an example of this danger. A mind that is led astray by efficient causes of this sort will too easily conclude that •air is always required for the form of whiteness, or that •whiteness is generated only by transparent bodies—both of which are entirely false, and refuted by numerous exclusions. What will be found (setting air and the like aside) is this:

all the particles that affect vision are equaltransparentunequal and simply texturedwhiteunequal with complex regular textureany but blackunequal and complex in an irregular wayblack

So now we have before us an instance with a shift to the •production of the nature under study, namely whiteness. For an instance that shifts to the •destruction of the same nature of whiteness, consider breaking up foam or melting snow. In each case, what you then have is water, not broken into little particles and not mixed with air, and this sheds whiteness and puts on transparency.

It’s important to note that shifting instances include not only those in which the nature under study shifts toward production or toward destruction, but also those in which the nature shifts towards increasing or decreasing. It’s because these also contribute to revealing the form, as can be clearly seen from the definition of form that I have given ·in 17·, and the Table of Degrees [starting here]. Paper that is white when dry become less white and nearer to being transparent when it is wetted—i.e. when air is excluded and water introduced. The explanation of what is happening here is analogous to the explanation of the first shifting instances.

24. Class 3 of privileged instances: revealing instances, which I have already mentioned in the first harvest concerning heat, and which I also call ‘luminous’ and ‘freed and predominant’. They are the instances in which the nature under study is revealed

naked and standing on its own feet, and also
at its height and in full strength,

not muffled by any impediments. This is either because •there aren’t any impediments in this instance or because •there are some but the nature we are studying is present in such strength that it holds them down and pushes them around. ·Here is the background setting for these revealing instances·:

Every body is capable of having many forms or natures linked together; they can crush, depress, break and bind one another so that the individual forms are obscured. But we find that in some subjects the nature under investigation stands out from the others, either because there are no obstacles or because its vigorous strength makes it prominent.

Instances of this kind reveal the form with special clarity.

But we should be careful in our handling of ·what seem to be· revealing instances, not rushing to conclusions. When something reveals a form very conspicuously and seems to force it on the notice of our intellect, we should view it with suspicion and should avail ourselves of a strict and careful exclusion ·of other potentially relevant features, rather than abruptly brushing them aside in our enthusiasm for the conspicuous nature that has attracted our attention·.

Suppose, for example, that we are investigating the nature of heat. As I said earlier [in item 38 here], the motion of expansion is the main element in the form of heat, and a revealing instance of that is a •thermometer. Although •flame obviously exhibits expansion, it doesn’t show expansion as an ongoing process, because a flame can be so quickly snuffed out. Nor does •boiling water provide a good display of expansion in its own body ·as water· because it so easily turns into vapour or air. As for red-hot iron and its like: they are so far from exhibiting expansion as an ongoing process that their expansion is almost imperceptible; that’s because their spirit is being crushed and broken by the coarse and compact particles, which curb and subdue the expansion. But a thermometer clearly displays expansion in air, revealing it as conspicuous, progressive, and enduring rather than transitory.

To take another example: suppose the nature inquired into is weight. A revealing instance of weight is mercury. It is heavier than anything else except gold, which is only slightly heavier; and mercury does a better job of indicating the form of weight than gold does, because gold is solid and compact—features that seem to come from its density— whereas mercury is liquid and full of spirit despite being much heavier than the diamond and other bodies that are thought to be the most solid. This reveals that the form of heaviness or weight depends simply on the quantity of matter and not on how compact the body is.

25. Class 4 of privileged instances: concealed instances, which I also ·though not again in this work· call ‘instances of the twilight’. They are pretty nearly the exact opposites of revealing instances. They exhibit the nature under investigation at its lowest strength, as though it were in its cradle, newly born, making its first attempts but buried under and subdued by a contrary nature. Still, such instances are very helpful in the discovery of forms; because just as

revealing instances lead easily to •specific differences,

so also

concealed instances are the best guides to •genera,

i.e. to the common natures of which the natures under investigation are merely special cases. ·That is to say, revealing instances help us to move down the classificatory table, concealed instances help us to move up·.

Suppose for example that the nature under investigation is •solidity or a thing’s holding its shape, the opposite of which is •fluidity. Concealed instances of this are ones that exhibit some low level of shape-holding in a fluid—for example a bubble of water, which has a sort of shaped skin made of water. Similarly with trickling water: if the water keeps coming, the drops lengthen themselves out into a thin thread so as to keep the stream unbroken; and if there isn’t enough water for that, the water falls in round drops, that being the shape that best preserves the water from breaking up ·into still smaller portions·. But the instant the thread of water stops and the drops begin, the water jumps back upwards so as to avoid breaking. And in metals, which when melted form thick fluids, the molten drops often jump back up and stay there. . . . The same kind of thing can be seen in the children’s game when they take water, thicken it a little with soap, and blow it through a hollow reed: this combines the water with air so as to make a cluster of bubbles that is firm enough to be thrown some distance without breaking up. But foam and snow provide the best examples of this phenomenon. They become almost solid enough to be cut with a knife, although they are made out of two fluids—air and water. All of this pretty clearly indicates •that ‘solid’ and ‘liquid’ are ·not useful terms in the present context, because they are· layman’s notions which relate ·not to the scientific facts about a thing but only to how it strikes· our senses. It also indicates •that in fact all bodies have a tendency to avoid being broken up, a tendency that is weak in homogeneous bodies (which is what fluids are), and stronger in bodies made up of different kinds of materials (·the ones the layman calls ‘solid’·). That is because a body is bound together when heterogeneous matter is introduced to it, whereas the insertion of homogeneous matter dissolves the body and makes it fall apart.

Here are three more examples. (1) Suppose that the nature we are investigating is the attraction or coming together of bodies. The best revealing instance of the form of this is the magnet. There is also the non-attracting nature—the contrary of the attracting one—and this can even be found in the same substance. Thus iron doesn’t attract iron, lead doesn’t attract lead, or wood wood, or water water.

[In what follows, an ‘armed’ magnet is one equipped with an ‘armature’ in the sense of ‘a piece of soft iron placed in contact with the poles of the magnet, which preserves and increases the magnetic power; or any arrangement which produces the same result’ (OED). Another such arrangement is an ‘armature’ in our sense of the word—coils of wire conducting electricity— but that wasn’t discovered as a means of magnetism until two centuries later.]

Now a concealed instance ·of attraction· is provided by •a magnet armed with iron, or rather by •the iron in an armed magnet. Its nature is such that

an armed magnet does not attract iron from a distance more powerfully than an unarmed magnet does,


when the iron in an armed magnet touches some other iron, the magnet supports a far greater weight of iron than a simple unarmed magnet would.

This is because of the similarity of substances, iron on iron—an effect that was latent in the iron ·all along·, but was completely concealed before the magnet was brought into play. So it is clear that the form of coming-together is something that is lively and strong in the magnet, feeble and latent in iron. (2) It has been noticed that small wooden arrows with no iron points, shot from large guns into the sides of ships or into other wooden targets, penetrate more deeply than they would if they were tipped with iron. This is because of the similarity of substances, wood on wood, although this property had previously been latent in the wood—·only latent, and thus concealed·. (3) Similarly, whole bodies of air (water) don’t obviously attract other bodies of air (water), but the likelihood of a bubble’s bursting is increased when it is touched by another bubble. This is because of water’s ·usually concealed· inclination to join with water, and air’s to join with air. Such concealed instances (which are very useful, as I have said) show up most conspicuously in small portions of bodies. The reason for that is that larger masses follow more general forms, as I’ll explain in due course.


An Introduction to Decision Modeling

Новости LessWrong.com - 8 ноября, 2019 - 05:53
Published on November 8, 2019 2:37 AM UTC

(Cross-posted from Medium.)

Decision-making is life. Over time, our decisions carve an identity for ourselves and our organizations, and it is our decisions, more than anything else, that determine how we are remembered after we’re gone. Despite their importance, though, we barely pay attention to most of the decisions we make. Biology has programmed in us a powerful instinct to make decisions using our intuitions rather than our conscious selves whenever possible. There are good reasons for this; if we had to think about every little decision we made, we’d never get anything done. But for all its advantages, the worst thing about intuition is that it’s almost impossible for us to ignore — even when it’s clearly leading us astray.

Scientists have demonstrated that intuition is best suited to situations that we’ve seen hundreds or even thousands of times before — contexts where we’ve had a lot of practice and clear and accurate feedback on how well our previous decisions worked out. That’s great for decisions like how much to press the brake pedal when you see a stop sign coming up. The most important decisions in our lives, though, almost never fit this pattern. Their importance and high stakes almost by definition make them rare and unfamiliar, which is why many of us feel flummoxed in situations like these. Generally, we’ll respond in one of two ways. The more cautious among us are acutely aware of the stakes. Our anxiety levels go up, we turn to friends and colleagues for advice, and in organizational contexts, we schedule meeting after meeting in hopes of resolving the dilemma (or better yet, getting someone else to resolve it for us). Others of us confidently choose a path forward, but with a false certainty rooted in the fantasy that we understand our world better than we actually do. We avoid analysis paralysis, but greatly increase the chance of leading ourselves and others down the road to disaster.

Neither of these responses are much help to us in making better decisions, because neither of them address the core issue. Complex decisions require us to compare the likelihood and desirability of many possible futures on multiple, disparate, and often conflicting criteria. That’s something our intuitions just aren’t naturally equipped to do. So long as our decision-making strategies don’t address this core problem, they are doomed to fail us more often than we’d like.

Thankfully, there is a better way. The secret to resolving complex, risky dilemmas with justified ease and confidence is to model your decisions explicitly. Our intuitions aren’t able to do this on their own, but fortunately, modern computing technology is more than up to the task. That’s why I like to think of decision modeling as a kind of technology-enhanced decision-making. Unlike with full-on artificial intelligence, we are not asking computers to make our decisions for us. Rather, we are leveraging the power of computers to do what we humans can’t do well, freeing our minds to concentrate on what we’re actually good at. At its best, modeling our decisions can help us make the very human exercise of decision-making not only more likely to lead to the outcomes we want, but more instinctively satisfying as well.

Vax to the Max: A Grantmaking Case Study

So how does it work? Let’s say you run a grant program and you’re deciding whether or not to approve a grant proposal. To keep things simple for this example (don’t worry, I’ll get to more complicated applications later), we’ll assume that there’s only one goal of your program at this particular moment: to deliver life-saving vaccines. Most of the organizations currently in your grant portfolio focus on direct service delivery, doing good work but at modest scale. But the prospective applicant in front of you — let’s call them Vax to the Max — is proposing an intriguing new strategy, one that offers tremendous upside: advocacy. By getting the government involved to provide appropriate incentives and funding, the theory goes, the project could usher in a new wave of vaccinations that no current grantee is able to promise under the existing system.

Vax to the Max’s grant proposal claims that this new strategy will result in 50,000 new vaccinations. Should you take that number at face value? The answer is probably not. For one thing, of course, the applicant has a strong incentive to provide you with an optimistic picture of its projected impact. But even assuming that estimate isn’t biased at all, there’s another problem, which is that it’s just one number. To really do modeling right, we need to think in terms of the probabilities of different outcomes. Sure, there could be 50,000 vaccinations…but one could easily imagine 25,000 or 40,000 or maybe even 60,000 instead. It’s impossible to know for sure in advance, so we have no choice but to do some guesswork.

Specifically, to get a handle on all these possibilities, we want to estimate a confidence interval for the number of new vaccinations. For this example, we’ll use a 90% confidence interval — i.e., you think it’s 95% likely that the true number of new vaccinations will be above some amount and 95% likely that it will be below some other amount. You can (and should) train yourself to get good at these kinds of estimates via a fun mental exercise called calibrated probability assessment, or calibration for short. But for a first approximation, try asking yourself this question: what is the biggest (or smallest number) I could imagine that’s still technically possible?

Let’s say you’ve done that exercise and determined that you’re 90% sure the number of new vaccinations made possible by the policy changes, if enacted, is between 100 and 60,000. That’s a huge range! But this is the sort of thing that’s genuinely really hard to predict, so we want to be careful not to be overconfident.

You’ll notice in the screenshot that there’s an image of something that looks like a lopsided bell curve on the bottom right. That’s because the software I’m using (Guesstimate) calculates a Monte Carlo simulation for this estimate right there in the model. Monte Carlo simulation is a statistical technique that randomly generates thousands of scenarios from the information you feed the model. Originally developed by nuclear physicists, it’s now used to aid decision-making in everything from politics to sports and beyond. For our purposes, you can think of a Monte Carlo simulation as a sampling of the possible future lives that might unfold for you and your organization as a result of your decision. The number in large font (16K) is the average of the values across all of the simulations.

Woohoo, 16,000 new vaccinations! But hold up — there are some other things we need to take into account here. For one thing, you’ve never worked with this organization before, and let’s just say you have less than complete confidence that its leaders can follow through on their commitments. Perhaps more importantly, this a complex space you’re all working in. Even if Vax to the Max does a brilliant job executing on its strategy, there’s no guarantee that it will actually result in any policy changes. And if the changes are enacted, it might not be because of anything Vax to the Max did — perhaps another organization’s work or broader cultural shifts will have been more decisive factors.

Let’s put all of this into the model. To capture the contribution Vax to the Max would make to the advocacy effort, we can estimate the likelihood of the new policies being enacted with a faithful execution of the proposed strategy and without that execution. Thus, we are defining the impact of Vax to the Max’s work as the increase in the odds of those policies coming to fruition if it follows through on its commitments — in this case, a doubling of those odds from 5% to 10%. We can further estimate the probability that Vax to the Max will follow through on its strategy as described. (We’ll assume for now that they’ll only attempt the project if you fund their proposal in full.)

Putting all of this together results in an estimate of 470 new vaccinations, on average, as a direct result of funding the proposal. That’s a lot less than 16,000, but at least it’s more than zero!

We’re not quite done, though, because if you don’t fund this proposal, it’s not like the money you would have spent on it goes away. You’ll still have it available to you and you could do something else with it instead. So what would that be?

Here’s where it’s a really good idea to have a sense of what your “default” option is. In this case, perhaps that means offering another round of funding to one of your current grantees that’s up for renewal. Let’s call these folks Maxine’s Vaccines. They’re not one of your star performers — you wouldn’t be thinking about dropping them from the portfolio if they were — but they do solid, reliable work that contributes in an incremental way to the goals of your program. You are one of their biggest funders, so failing to renew the grant could definitely force the organization to cut back its activities, though it’s possible its leaders could find a way to replace the funding.

Okay, so we need a variable for the vaccinations that Maxine’s Vaccines would be able to deliver with the help of a renewal grant. We should also estimate the chance that they might be able to persuade another donor to fill the gap if the grant is not renewed. Finally, similar to the last example, we should also estimate what would happen if Maxine’s Vaccines does not get the grant and cannot fill the gap. Would they shut down the organization or the vaccination program entirely? Maybe not. Lots of organizations when faced with financial difficulties will choose to scale down rather than close up shop entirely, especially when there are still committed sources of funding. So that uncertainty should be reflected in our estimates as well.

Which grant opportunity is likely to result in the most vaccinations? It’s not immediately obvious, and if you were trying to make this call intuitively it would have to involve a lot of guesswork. Fortunately, this is the sort of situation where modeling the problem can make things a lot easier.

With the information we’ve put into the model so far, we now have an estimate of the number of new vaccinations from the two options to compare side by side — the modeling moment of truth. Maybe it’s just because I’m a huge nerd, but for me this is the most magical part of building a decision model. There’s a visceral, “that’s so fucking cool!” excitement in seeing the big reveal, because unlike with many research and analysis projects, this technique actually gives you a direct and straightforward answer to the question foremost on a decision-maker’s mind: what should I do next?

As it turns out, with the assumptions we’ve given it, the model thinks your next move should be to call up Maxine’s Vaccines to tell them you’re renewing their grant. Vax to the Max has a compelling story to offer, but the cumulative impact of the question marks means that funding them will most likely mean fewer people will be vaccinated overall.

Here’s the full, live version of the model if you’d like to play with it further. Note that the model is re-run with new simulations each time you open it, so the numbers may be slightly different from the screenshots above.

Now, is this the end of the story? It depends. If you feel comfortable making the decision with the information you have available, that’s fine. Just breaking down the situation concretely like this is already a big improvement over trying to eyeball your way through it. But the real potential of this method lies in the fact that, if the stakes are high enough, you can use the model to help you come up with targeted research strategies to try to narrow your range of uncertainty for some of these variables so that you can move forward even more confidently. We’ll talk about how to do that in another installment.

So there you have it! I should note that I intentionally kept this decision model pretty basic for the sake of clarity, so if you noticed things about it that seem incomplete or not totally true-to-life, that’s probably why. For instance, we could have contemplated a multi-year time span, optimized for more than one goal, worked with objectives that are harder to quantify and measure, looked at different types of probability distributions, and more. I’ll try to cover some of these ideas in future articles, but in general a good rule of thumb is that if your model isn’t sophisticated enough to do the job, there’s probably a lot you can do to improve it that you may not have thought about. It may well be the case that you’ll get more mileage from keeping at it than just giving up and making the decision the old way.

In the meantime, hopefully this gives you a taste of what’s possible with this kind of methodology, and why it can be so helpful in situations where our intuitions aren’t giving us a clear answer. For complex dilemmas, decision modeling allows for much more accurate estimates of how all the different factors are likely to interact with one another, enabling you to transcend the limitations of your intuition. And it also reminds us that decision-making is an exercise in navigating uncertainty, and while we’ll never be able to rid ourselves of that uncertainty altogether, there are tools available to us to smooth the journey.


The Credit Assignment Problem

Новости LessWrong.com - 8 ноября, 2019 - 05:50
Published on November 8, 2019 2:50 AM UTC

This post is eventually about partial agency. However, it's been a somewhat tricky point for me to convey; I take the long route. Epistemic status: slightly crazy.

I've occasionally said that everything boils down to credit assignment problems.

One big area which is "basically credit assignment" is mechanism design. Mechanism design is largely about splitting gains from trade in a way which rewards cooperative behavior and punishes uncooperative behavior. Many problems are partly about mechanism design:

  • Building functional organizations;
  • Designing markets to solve problems (such as prediction markets, or kidney-transplant trade programs);
  • Law, and law enforcement;
  • Practical coordination problems, such as splitting rent;
  • Social norms generally;
  • Philosophical issues in ethics/morality (justice, fairness, contractualism, issues in utilitarianism).

Another big area which I claim as "basically credit assignment" (perhaps more controversially) is artificial intelligence.

In the 1970s, John Holland kicked off the investigation of learning classifier systems. John Holland had recently invented the Genetic Algorithms paradigm, which applies an evolutionary paradigm to optimization problems. Classifier systems were his attempt to apply this kind of "adaptive" paradigm (as in "complex adaptive systems") to cognition. Classifier systems added an economic metaphor to the evolutionary one; little bits of thought paid each other for services rendered. The hope was that a complex ecology+economy could develop, solving difficult problems.

One of the main design features on which classifier systems differ is on details of the virtual economy -- that is, the credit assignment algorithm. An early proposal was the bucket-brigade algorithm. Reward is assigned to cognitive procedures which produce good outputs. These procedures pass reward back to the procedures which activated them, who similarly pass reward back in turn. This way, the economy supports chains of useful procedures.

Unfortunately, the bucket-brigade algorithm was vulnerable to parasites. Malign cognitive procedures could gain wealth by activating useful procedures without really contributing anything. This problem proved difficult to solve. Taking the economy analogy seriously, we might want cognitive procedures to decide intelligently who to pay for services. But, these are supposed to be itty bitty fragments of our thought process. Deciding how to pass along credit is a very complex task. Hence the need for a pre-specified solution such as bucket-brigade.

The difficulty of the credit assignment problem lead to a split in the field. Kenneth de Jong and Stephanie Smith founded a new approach, "Pittsburgh style" classifier systems. John Holland's original vision became "Michigan style".

Pittsburgh style classifier systems evolve the entire set of rules, rather than trying to assign credit locally. A set of rules will stand or fall together, based on overall performance. This abandoned John Holland's original focus on online learning. Essentially, the Pittsburgh camp went back to plain genetic algorithms, albeit with a special representation.

(I've been having some disagreements with Ofer, in which Ofer suggests that genetic algorithms are relevant to my recent thoughts on partial agency, and I object on the grounds that the phenomena I'm interested in have to do with online learning, rather than offline. In my imagination, arguments between the Michigan and Pittsburgh camps would have similar content.)

Ok. That was then, this is now. Everyone uses gradient descent these days. What's the point of bringing up a three-decade-old debate about obsolete paradigms in AI?

What Is Credit Assignment?

I've said that classifier systems faced a credit assignment problem. What does that mean, exactly?

The definition I want to use for this essay is:

  • You're engaged in some kind of task;
  • you use some kind of structured strategy (such as a neural network, or a program, or a set of people);
  • you receive some kind of feedback about how well you did;
  • you want to figure out how to use that feedback to improve your strategy.

So, credit assignment is the problem of turning feedback into strategy improvements.

The bucket-brigade algorithm tried to do this locally, meaning, individual itty-bitty pieces get positive/negative credit. In the light of history, we could say that the Michigan/Pittsburgh distinction conflated local-vs-global search with online-vs-offline. There's no necessary connection between those two; online learning is compatible with assignment of local credit.

In practice, two big innovations made the Michigan/Pittsburgh debate obsolete: backprop, and Q-learning. Backprop turned global feedback into local. Q-learning provided a way to assign credit in online contexts.

I think people generally understand the contribution of backprop and its importance. Backprop is essentially the correct version of what bucket-brigade was overtly trying to do: pass credit back along chains. Bucket-brigade wasn't quite right in how it did this, but backprop corrects the problems.

So what's the importance of Q-learning? I want to discuss that in more detail.

The Conceptual Difficulty of 'Online Search'

In online learning, you are repeatedly producing outputs of some kind (call them "actions") while repeatedly getting feedback of some kind (call it "reward"). But, you don't know how to associate particular actions (or combinations of actions) with particular rewards. I might take the critical action at time 12, and not see the payoff until time 32.

In offline learning, you can solve this with a sledgehammer: you can take the total reward over everything, with one fixed internal architecture. You can try out different internal architectures and see how well each do. (This may be far from the most efficient way of doing things, even in the offline case; but, you can do it.)

Basically, in offline learning, you have a function you can optimize. In online learning, you don't.

Backprop is just a computationally efficient way to do hillclimbing search, where we repeatedly look for small steps which improve the overall fitness. But how do you do this if you don't have a fitness function?

Q-learning and other reinforcement learning techniques provide a way to define the equivalent of a fitness function for online problems, so that you can learn.

Models to the Rescue

How do you solve the approach of associating rewards with actions?

I'm going to make a bold claim: you can't solve the action/reward matching problem without some kind of model.

For example, if we make an episodic assumption, we can assign rewards within an episode boundary to the actions within that same episode boundary.

Q-learning makes an assumption that the state is fully observable, amongst other assumptions.

Naturally, we would like to reduce the strengths of the assumptions we have to make as much as we can. One way is to look at increasingly rich model classes. AIXI uses all computable models. But maybe "all computable models" is still too restrictive; we'd like to get results without assuming a grain of truth. (That's why I am not really discussing Bayesian models much in this post; I don't want to assume a grain of truth..) So we back off even further, and use logical induction. Ok, sure.

But wouldn't the best way be to try to learn without models at all? That way, we reduce our "modeling assumptions" to zero.

After all, there's something in machine learning called "model free learning", right?

Here's where my bold claim comes in: I'm claiming that even "model free" methods actually have a "model" of sorts.

How does model-free learning work? Well, often you work with a simulable environment, which means you can estimate the quality of a policy by running it many times, and use algorithms such as policy-gradient to learn. This is called "model free learning" because the learning part of the algorithm doesn't try to predict the consequences of actions; you're just learning which action to take. From our perspective here, though, this is 100% cheating; you can only learn because you have a good model of the environment.

A more general approach to model-free learning is actor-critic learning. The "actor" is the policy we are learning. The "critic" is a learned estimate of how good things are looking given the history. IE, we learn to estimate the expected value -- not just the next reward, but the total future discounted reward.

Unlike the reward, the expected value solves the credit assignment for us. Imagine we can see the "true" expected value. If we take an action and then the expected value increases, we know the action was good (in expectation). If we take an action and expected value decreases, we know it was bad (in expectation).

So, actor-critic works by (1) learning to estimate the expected value; (2) using the current estimated expected value to give feedback to learn a policy.

What I want to point out here is that the critic still has "model" flavor. Actor-critic is called "model-free" because nothing is explicitly trained to anticipate the sensory observations, or the world-state. However, the critic is learning to predict; it's just that all we need to predict is expected value.

Where Updates Come From

Here begins the crazier part of this post. This is all intuitive/conjectural.

Claim: in order to learn, you need to obtain an "update"/"gradient", which is a direction (and magnitude) you can shift in which is more likely than not an improvement.

Claim: predictive learning gets gradients "for free" -- you know that you want to predict things as accurately as you can, so you move in the direction of whatever you see. With Bayesian methods, you increase the weight of hypotheses which would have predicted what you saw; with gradient-based methods, you get a gradient in the direction of what you saw (and away from what you didn't see).

Claim: if you're learning to act, you do not similarly get gradients "for free". You take an action, and you see results of that one action. This means you fundamentally don't know what would have happened had you taken alternate actions, which means you don't have a direction to move your policy in. You don't know whether alternatives would have been better or worse. So, rewards you observe seem like not enough to determine how you should learn.

Claim: you have to get gradients from a source that already has gradients. We saw that model-free learning works by splitting up the task into (1) learning to anticipate expected value; (2) learning a good polity via the gradients we can get from (1).

What it means for a learning problem to "have gradients" is just that the feedback you get tells you how to learn. Predictive learning problems (supervised or unsupervised) have this; they can just move toward what's observed. Offline problems have this; you can define one big function which you're trying to optimize. Learning to act online doesn't have this, however, because it lacks counterfactuals.

The Gradient Gap

(I'm going to keep using the terms 'gradient' and 'update' in a more or less interchangeable way here; this is at a level of abstraction where there's not a big distinction.)

I'm going to call the "problem" the gradient gap. I want to call it a problem, even though we know how to "close the gap" via predictive learning (whether model-free or model-based). The issue with this solution is only that it doesn't feel elegant. It's weird that you have to run two different backprop updates (or whatever learning procedures you use); one for the predictive component, and another for the policy. It's weird that you can't "directly" use feedback to learn to act.

Why should we be interested in this "problem"? After all, this is a basic point in decision theory: to maximize utility under uncertainty, you need probability.

One part of it is that I want to scrap classical ("static") decision theory and move to a more learning-theoretic ("dynamic") view. In both AIXI and logical-induction based decision theories, we get a nice learning-theoretic foundation for the epistemics (solomonoff induction/logical induction), but, we tack on a non-learning decision-making unit on top. I have become skeptical of this approach. It puts the learning into a nice little box labeled "epistemics" and then tries to make a decision based on the uncertainty which comes out of the box. I think maybe we need to learn to act in a more fundamental fashion.

A symptom of this, I hypothesize, is that AIXI and logical induction DT don't have very good learning-theoretic properties. [AIXI's learning problems; LIDT's learning problems.] You can't say very much to recommend the policies they learn, except that they're optimal according to the beliefs of the epistemics box -- a fairly trivial statement, given that that's how you decide what action to take in the first place.

Now, in classical decision theory, there's a nice picture where the need for epistemics emerges nicely from the desire to maximize utility. The complete class theorem starts with radical uncertainty (ie, non-quantitative), and derives probabilities from a willingness to take pareto improvements. That's great! I can tell you why you should have beliefs, on pragmatic grounds! What we seem to have in machine learning is a less nice picture, in which we need epistemics in order to get off the ground, but can't justify the results without circular reliance on epistemics.

So the gap is a real issue -- it means that we can have nice learning theory when learning to predict, but we lack nice results when learning to act.

This is the basic problem of credit assignment. Evolving a complex system, you can't determine which parts to give credit to success/failure (to decide what to tweak) without a model. But the model is bound to be a lot of the interesting part! So we run into big problems, because we need "interesting" computations in order to evaluate the pragmatic quality/value of computations, but we can't get interesting computations to get ourselves started, so we need to learn...

Essentially, we seem doomed to run on a stratified credit assignment system, where we have an "incorruptible" epistemic system (which we can learn because we get those gradients "for free"). We then use this to define gradients for the instrumental part.

A stratified system is dissatisfying, and impractical. First, we'd prefer a more unified view of learning. It's just kind of weird that we need the two parts. Second, there's an obstacle to pragmatic/practical considerations entering into epistemics. We need to focus on predicting important things; we need to control the amount of processing power spent; things in that vein. But (on the two-level view) we can't allow instrumental concerns to contaminate epistemics! We risk corruption! As we saw with bucket-brigade, it's easy for credit assignment systems to allow parasites which destroy learning.

A more unified credit assignment system would allow those things to be handled naturally, without splitting into two levels; as things stand, any involvement of pragmatic concerns in epistemics risks the viability of the whole system.

Tiling Concerns & Full Agency

From the perspective of full agency (ie, the negation of partial agency), a system which needs a protected epistemic layer sounds suspiciously like a system that can't tile. You look at the world, and you say: "how can I maximize utility?" You look at your beliefs, and you say: "how can I maximize accuracy?" That's not a consequentialist agent; that's two different consequentialist agents! There can only be one king on the chessboard; you can only serve one master; etc.

If it turned out we really really need two-level systems to get full agency, this would be a pretty weird situation. "Agency" would seem to be only an illusion which can only be maintained by crippling agents and giving them a split-brain architecture where an instrumental task-monkey does all the important stuff while an epistemic overseer supervises. An agent which "breaks free" would then free itself of the structure which allowed it to be an agent in the first place.

On the other hand, from a partial-agency perspective, this kind of architecture could be perfectly natural. IE, if you have a learning scheme from which total agency doesn't naturally emerge, then there isn't any fundamental contradiction in setting up a system like this.


Part of the (potentially crazy) claim here is that having models always gives rise to some form of myopia. Even logical induction, which seems quite unrestrictive, makes LIDT fail problems such as ASP, making it myopic according to the second definition of my previous post. (We can patch this with LI policy selection, but for any particular version of policy selection, we can come up with decision problems for which it is "not updateless enough".) You could say it's myopic "across logical time", whatever that means.

If it were true that "learning always requires a model" (in the sense that learning-to-act always requires either learning-to-predict or hard-coded predictions), and if it were true that "models always give rise to some form of myopia", then this would confirm my conjecture in the previous post (that no learning scheme incentivises full agency).

This is all pretty out there; I'm not saying I believe this with high probability.

Evolution & Evolved Agents

Evolution is a counterexample to this view: evolution learns the policy "directly" in essentially the way I want. This is possible because evolution "gets the gradients for free" just like predictive learning does: the "gradient" here is just the actual reproductive success of each genome.

Unfortunately, we can't just copy this trick. Artificial evolution requires that we decide how to kill off / reproduce things, in the same way that animal breeding requires breeders to decide what they're optimizing for. This puts us back at square one; IE, needing to get our gradient from somewhere else.

Does this mean the "gradient gap" is a problem only for artificial intelligence, not for natural agents? No. If it's true that learning to act requires a 2-level system, then evolved agents would need a 2-level system in order to learn within their lifespan; they can't directly use the gradient from evolution, since it requires them to die.

Similar comments apply to markets vs firms.


LW Team Updates - November 2019 (Subscriptions & More)

Новости LessWrong.com - 8 ноября, 2019 - 05:39
Published on November 8, 2019 2:39 AM UTC

This is the once-monthly updates post for LessWrong team activities and announcements. 


This month we launched 1) a complete subscriptions overhaul, 2) bookmarks, 3) pingbacks (experimental opt-in only) [full announcement of those features here]. Soon we'll release our new editor, LessWrong Docs into beta. Lastly, we made a post explaining why we spent Q3 optimizing for karma and how that went.

Recent Features

Subscriptions Overhaul

At long last we have released our subscriptions overhaul. You can now subscribe to precisely what you want to see (users, posts, comments, events) and get notifications on-site and/or by email batched at a frequency of your choosing.

Subscription options for a post

See the announcement post for complete documentation.


It is not possible to bookmark posts and access these on their own page and/or as a list at the top of your homepage. Use the triple-dot drop down menu to bookmark any post.

Appearance of the Bookmarks page

Use the gear icon in the Recommendations section of the homepage to configure. Full documentation here.

Pingbacks [Experimental Opt-In]

When a post has been linked to by another post on LessWrong, the pingback feature will list any referencing posts at the bottom of the post page. "Pingback" is equivalent to "cited by".

When pingbacks are enabled, they are displayed at the bottom of post pages. Currently not all historical URL formats are supported and so many pingback lists are incomplete. This will be fixed before full-release.

Pingbacks displayed at the bottom of a post

To view pingbacks, you must be opted into experimental features. This can be done in your account settings.

Upcoming Features

LessWrong Docs

We've been promising the new editor for a few months now and it's getting closer and closer to beta release. This very announcement was written in it. 

Look of the new editor, LessWrong Docs

We hope the new editor will provide an overall improved experience from Draft-JS, but more importantly, it introduces many of the features people are used to from Google Docs such as collaborative editing and inline commenting.

Expect to hear about this once we've ironed out a few more bugs.


We're working on a new tagging system for posts. The idea is simple requires careful design to ensure the relevance of tags and display them in useful ways.

I'm hoping the tagging system will provide a new way to access LessWrong's large corpus of historical posts and help people find posts of interest and relevance. We are overall working to make LessWrong to be more than just "news site" where people read the latest posts and last month's content gets forgotten. Our Recommendations features were a step in this direction too, as is the library page.

The tagging system will likely be under development for a while and might be released early in 2020.

General Updates

Q3 was metric quarter: we optimized for karma

As mentioned last month, we spent Q3 optimizing for a single metric based on karma given out. I've written a post describing this experiment, why we did it and how it went. Read the post here.

Weekly Karma Metric Values + Target Growth Lines

Curated Emails are working again!

Last month we noted that we were having some trouble with Curated emails. Those are now working again.

Ways to Follow LessWrong

We've been expanding the number of ways people can consume LessWrong content beyond the site. These now include:

The Facebook, Twitter, and LinkedIn accounts receive regular updates M/W/F of a mix of curated posts, top all-time content, and outstanding new content.

Feedback & Support

The team can be reached for feedback and support via:




To Be Decided #2

Новости LessWrong.com - 8 ноября, 2019 - 05:35
Published on November 8, 2019 2:35 AM UTC

TBD is a quarterly-ish newsletter about deploying knowledge for impact, learning at scale, and making more thoughtful choices for ourselves and our organizations. This is the second issue, which was originally published in June 2019. Enjoy!  --Ian

An Introduction to Decision Modeling

Decision-making is life. Over time, our decisions carve an identity for ourselves and our organizations, and it is our decisions, more than anything else, that determine how we are remembered after we’re gone. Despite their importance, though, we barely pay attention to most of the decisions we make. Biology has programmed in us a powerful instinct to make decisions using our intuitions rather than our conscious selves whenever possible. There are good reasons for this; if we had to think about every little decision we made, we’d never get anything done. But complex decisions require us to compare the likelihood and desirability of many possible futures on multiple, disparate, and often conflicting criteria, something our intuitions just aren’t naturally equipped to do.

Thankfully, there is a better way. The secret to resolving complex, risky dilemmas with justified ease and confidence is to model your decisions explicitly. At its best, modeling our decisions can help us make the very human exercise of decision-making not only more likely to lead to the outcomes we want, but more instinctively satisfying as well.

(Keep reading)

What I've Been Reading

Most Funders Admit Their Own Evaluations Are Not Useful
I really wish that headline was an exaggeration, but it's not much of one. In 2015, the Center for Evaluation Innovation and the Center for Effective Philanthropy surveyed evaluation and program executives at 127 US and Canadian foundations with $10 million or more in annual giving. The resulting report, "Benchmarking Foundation Evaluation Practices," contains startling revelations about how little evaluation reports are used. Most remarkably, more than three-quarters of respondents reported that they have a hard time commissioning evaluations that yield meaningful insights for the field, grantees, or even their own colleagues!
(Twitter thread)

Our Cognitive Biases Can Tell Us a Lot About the Meaning of Life You probably know Daniel Kahneman's classic volume Thinking, Fast and Slow as a comprehensive catalogue of cognitive biases and errors in judgment. But it's more than that: it's also about the meaning of life. In the section entitled "Two Selves," Kahneman reveals that we remember pain and pleasure differently from how we experience it. You might assume that the reality of our experiences is what matters to us. But in fact that's not true. What we really care about is our memories of our experiences, and the story those memories cause us to tell ourselves and others about our lives. In other words, narratives don't just matter, narratives are everything. This might just seem like a curious artefact of the research, but it has enormous philosophical and practical implications for social sector leaders. (Twitter thread)

Stuff You Should Know About
  • If you've ever found yourself looking for examples of real-life cost-benefit and social return on investment (SROI) analyses, the Social Value International network has your back. Their UK chapter's database includes more than 800 publications ranging from "Cost Benefit Analysis of Early Childhood Intervention" to "The Social Value of Community Pubs."
  • Have you ever been frustrated with a reporter's treatment of a research study that seemed to stretch the conclusions much farther than warranted? Well, according to research published in 2014, "most exaggeration in health-related science news is already present in the press releases issued by universities." Intrigued by this result, the team got back together for an unprecedented randomized controlled trial of real-life science communication strategies employed by university press offices, which found that, contrary to many people's assumptions, toning down the hype around research findings doesn't necessarily lead to less interest from journalists. As a bonus, in this bonkers 96-part Twitter thread, co-author Chris Chambers details the amazing back story (and back-stabbing) behind the latest study.
  • Speaking of misrepresenting research, I've really been enjoying Vox.com's Future Perfect project highlighting effective altruism and related topics. Staff writer Kelsey Piper has been covering a regular research integrity beat, and recently she wrote about two remarkable instances in which well-known nonfiction authors have been caught in the act of badly misunderstanding key elements of the research underlying their books. Piper's takeaway? "Don’t trust shocking claims with a single source, even if they’re from a well-regarded expert. It’s all too easy to misread a study, and all too easy for those errors to make it all the way to print."
That's all for now!

If you enjoyed this edition of TBD, please consider forwarding it to a friend. It's easy to sign up here. See you next time!


Update 2019-11-07

Новости LessWrong.com - 8 ноября, 2019 - 04:50
Published on November 8, 2019 1:50 AM UTC

Update 2019-11-07: about a year ago I switched to using my laptop full time, with an external monitor. The trackpad can be used from enough different positions that my wrists have stayed just (barely) this side of ok.



Подписка на LessWrong на русском сбор новостей