# Новости LessWrong.com

A community blog devoted to refining the art of rationality
Обновлено: 12 минут 37 секунд назад

### Examples of Causal Abstraction

13 декабря, 2019 - 01:54
Published on December 12, 2019 10:54 PM UTC

I’m working on a theory of abstraction suitable as a foundation for embedded agency and specifically multi-level world models. I want to use real-world examples to build a fast feedback loop for theory development, so a natural first step is to build a starting list of examples which capture various relevant aspects of the problem.

These are mainly focused on causal abstraction, in which both the concrete and abstract model are causal DAGs with some natural correspondence between counterfactuals on the two. (There are some exceptions, though.) The list isn’t very long; I’ve chosen a handful of representative examples which cover qualitatively different aspects of the general problem.

I’ve grouped the examples by symmetry class:

• Finite DAGs without any symmetries, or at least no symmetries which matter for our purposes
• Plate symmetry (as in “plate notation”), in which there are a number of conditionally IID components
• Time symmetry, in which the DAG (or some part of it) consists of one repeated subcomponent connected in a straight line (i.e. a Markov chain structure)

Note that many of the abstractions below abstract from one symmetry class to another - for example, MCMC abstracts a concrete time-symmetric model into an abstract plate-symmetric model.

I’m interested to hear more examples, especially examples which emphasize qualitative features which are absent from any of the examples here. Examples in which other symmetry classes play an important role are of particular interest, as well as examples with agenty behavior which we know how to formalize without too much mess.

Finite DAGs: Examples from Electrical Circuits

Electrical engineers rely heavily on nested layers of abstraction, of exactly the sort I’m interested in (i.e. multi-level models of the physical world). Additionally, causal models are a natural fit for digital circuits. These properties make electrical circuits ideal starting points. They’re a great conceptually-simple use case.

A few of the major abstraction layers, from lowest to highest:

• Fields: the most concrete-level model used in EE
• Lumped circuit abstraction: approximating the system as discrete “wires” with constant voltage connecting “circuit elements” with various voltage-current relationships and internal state.
• Digital abstraction: bucket voltages into high and low.
• Logic abstraction: replace various subcircuits with logic gates, and multiple “wires” with logical connections.
• Arithmetic abstraction: replace a logic circuit with an arithmetic circuit
• Floating point & modular arithmetic: throw out least-significant bits vs throw out most-significant bits
• Software-level abstractions, e.g. IP -> TCP -> HTTP

Note that real circuits usually do contain some repeated sub-components, but the symmetries in these DAGs aren’t particularly relevant to our purposes, so we’ll mostly ignore them.

Parallel to all this, somewhere along the way we usually abstract out the low-level continuous time-dependence, and adopt an abstract model of instantaneous input-output circuits coupled to clocked storage units (i.e. flip-flops/registers). We’ll include that abstraction separately in the time symmetry section; the levels from lumped circuit through floating point/modular arithmetic can all be specialized to memoryless input-output circuits for simplicity.

Plate Symmetry: Statistical Toy Models

This is the simplest nontrivial symmetry class. The main new qualitative phenomena I see in this class are:

• Nodes which attempt to estimate the value of other nodes, i.e. embedded maps/embedded reasoners. Technically we can have these in finite DAGs too, but they’re most natural to first consider in models with plate-symmetry, since that’s where traditional statistics operates.
• Two types of counterfactuals on symmetric components: those which act on only one component, and those which act symmetrically on all.
• The possibility that an embedded reasoner (i.e. statistical method) can leverage knowledge of the symmetry.

The use of sufficient statistics is a particularly simple example in this class, and adding the calculation of sufficient statistics as an explicit node in the DAG gives us the simplest embedded map. This is the easiest model I’ve used to ask questions like “when can we use the map in place of the territory?” - i.e. questions about abstractions embedded in the DAG itself.

Another example of interest in this class is an embedded reasoner which attempts to deduce model structure by leveraging symmetry. In particular, this introduces the possibility that a node in the DAG could detect (some) counterfactual modifications of the DAG - i.e. notice when it is in a counterfactual query.

Time Symmetry: Equilibrium -> Causality

This is the main symmetry class of interest at the level of physics for most systems, so there’s a lot of examples. Most of them involve some kind of equilibrium abstraction: the concrete model is a DAG over time, while the abstract model captures long-run behavior with time removed.

The simplest example is circuit equilibrium, which we mentioned earlier. At the physical level, the behavior of electrical circuits is DAG-shaped only when viewed over time. Yet, in many applications, there are “inputs” and “outputs” and the equilibrium state of the electrical circuit implements a DAG of some sort. Where does the abstract causal structure come from? This problem is also very similar to causality arising in equilibrium in other areas, e.g. biochemical signalling circuits in cells, or markets/supply chains in which certain goods have very high/very low price elasticity.

The next simplest example is timescale separation, in which a part of the system equilibrates much faster than the rest. A couple examples in this class:

• Fast equilibrium approximations in chemical kinetics (leading to an abstract causal model in which production & removal rates are parents of equilibrium levels)
• Alternate updating of fast equilibrium & slow dynamics, e.g. flip-flops/registers paired with fast memoryless input-output circuits in digital electronics.

MCMC is a particularly interesting example. The baby version of this example is the independence of widely-time-separated samples from a markov chain; that’s a simple prototypical example of abstracting time-symmetry into plate-symmetry. But MCMC adds DAG structure within the plate, in a way which does not directly mirror the DAG structure of the concrete model (although it does mirror the undirected structure). It also involves probability calculations in each (concrete) node, which is a hint that an embedded map is present in the system.

Of course, looking at abstractions of time-symmetric systems, we can’t omit feedback control. Despite loopy behavior on the concrete level, at the abstract level we can view the controller target point as causing system limiting behavior - and this abstract view will correctly handle many counterfactuals. In this case, the structure of the abstract equilibrium model might not match the concrete-level structure at all. Based on the good regulator theorem, this is another case where embedded maps are likely to be involved.

Finally, one particularly difficult example: the derivation of the Navier-Stokes equations from molecular dynamics. The main qualitative difference from the earlier examples (at least that I know of) is the importance of an ontology shift: a move from particles to fields of delta functions, from Hamiltonian particle dynamics to Vlasov/Boltzmann equations. Without that shift, our DAG structure shifts over time - because interactions are spatially organized, particles interact with different particles depending on where they are. (Note that deriving Navier-Stokes from particle dynamics is arguably an open problem, depending on what exactly we count as a “derivation”, so there may be other interesting aspects to this example as well. Or possibly not - calculation difficulties, rather than fundamental/conceptual difficulties, seem to be considered the main blockade to a derivation.)

Discuss

### Moloch feeds on opportunity

13 декабря, 2019 - 00:05
Published on December 12, 2019 9:05 PM UTC

This is very likely my most important idea, and I've been trying to write about it for years, but it's too important to write about it badly, so I haven't ever mustered the courage. I guess I'll just do a quick test round.

It starts with this "hierarchy of needs" model, first popularized my Maslow, that we humans tend to focus on one need at the time.

• If you're hungry, you don't care about aesthetics that much
• If you are competing for status, you can easily be tempted to defect on coordination problems
• etc

I like to model this roughly as an ensemble of subagents, with a power ordering. Each subagent has their say. Some are much more powerful than others, but they won't make use of their power if they're satisfied, in which case less prominent subagents get to decide.

As in, you're walking on the street thinking about math, but then a bus nearly hits you, so your much more powerful safety-oriented subagent takes over control, and you forget about math for a while.

Or you're thinking about math because it's interesting, but then you have to compete for tenure, and your status-seeking career-survivalist subagent kicks in, and starts thinking about problems that are prestigious but not that interesting.

And so your subagents take turns, and it's always the turn of the subagent that is oriented towards the most prominent unsatisfied need.

I have thought a lot about what constitutes "satisfaction". At what point does a subagent let go of the reins because it's achieved what it wants? I've found a few different possibilities:

• Something biologically hardcoded. i.e. you're satiated from your meal, or it's warm enough, or whatever. This isn't a very interesting case.
• You run out of perceived opportunity.

In that sense, as I've said before, human beings really are satisficers. They will stop optimizing for something if they don't see any possible way to get more of it. They run out of perceived opportunity.

But here's the kicker: in this globalist hyper-connected century, we don't really run out of perceived opportunity anymore. What does happen, is that we're perpetually stuck with motivations that people in the past would have perceived as morally depraved. These are motivations for which there used to be much less to gain. Motivations for which everyone used to be satisfied, because there was much less to compare oneself with.

This is especially problematic when it comes to our subagent that perceives status.

Status has a bad rap for a good reason. It makes us compete. It cuts us off from good and healthy things. The subagent that pursues status is more prominent than the subagent that pursues that vague blob of things we call names like wholesomeness, happiness, love, and humanity (Maslow called it self-actualisation).So how do we stop the status subagent? By removing our opportunity for status altogether. I have the suspicion that healthy, high-trust cultures tend to make status as predictable and as hard to change as possible:

• We don't talk about money to reduce the incentive to make a lot of it for the sake of status.
• Healthy groups of friends tend to occasionally bully the most successful person in the group to keep everyone on the same level
• The institution of marriage is meant to lock in your romantic success indefinitely, so your sexually competitive subagent is forever silent.
• Many societies tend to award status based on seniority, or the amount of years you've been around. This reduces turnover, which is a proxy for opportunity, but it also makes it impossible to change your level of status. I've seen this in my fraternity, but also in my local buddhist temple.
• The stereotypical woman usually hates it when men brag about their conquests, because it creates an incentive to get laid for the wrong reasons. In other words, she would rather engage with someone who is optimizing for self-actualization as opposed to status.

All of this is some kind of reduction of (perceived) opportunity. More precisely, all of these things reduce the perceived reward of making active steps towards status. They kill the competition trigger.

If you're thinking about designing patterns that create cooperation, this idea should be front and center in your mind.

Moloch feeds on opportunity.

Discuss

12 декабря, 2019 - 06:30
Published on December 12, 2019 3:30 AM UTC

I like playing Go, but of the games I play at all regularly it's probably the one where there are the most people who are better than me: a lot of people play go, and many of them play it a lot.

My dad and I are a similar level (~11kyu) and last week a family friend who's substantially better (~6kyu) was looking for a game. Instead of either of us playing him 1:1 with ~5 handicap stones, we decided to play 2:1 even. As a casual game, we played untimed.

It was a really interesting game: my dad any I both compensated for each others weaknesses, were more thoughtful and careful, didn't make any major mistakes, and ended up winning by a lot. It felt a lot like pair programming.

I'm very curious now how much better playing as a pair tends to make people. While I expected us to do better, +5 ranks (+500 ELO) was a much larger change than I would have guessed. It also may not be linear: I would expect better players to generally benefit less, since worse players are more likely to have uneven-but-complementary skills?

Discuss

### The Cybersecurity Dilemma in a Nutshell

12 декабря, 2019 - 06:20
Published on December 12, 2019 3:20 AM UTC

The Cybersecurity Dilemma: Hacking, Trust, and Fear Between Nations organizes its arguments in a fairly no-nonsense, premise –> premise –> conclusion manner that I thought would be good to summarize with a blog post. You can buy the book here: https://www.amazon.com/Cybersecurity-Dilemma-Hacking-Between-Nations-ebook/dp/B06W2L7K6X

The argument:

Security dilemmas are dangerous because in seeking their own security, states often build capabilities and take actions that can directly threaten the security of other states: often creating impressions of imminent offensive intent and prompting escalation. This is especially true for cybersecurity for several main reasons (Click on the triangles for more details):

States that desire options for future cyber operations must make intrusions in advance.
• The sequential stages of cyber-attack (target acquisition, development, authorization, entry, establishing command and control (C2), pivoting, payload activation, and confirmation) can take a very long time.
• E.g. time required to find zero-days, develop exploits and new tools, get authorization, get past defenses such as airgaps, set up C2 and to make pivots within networks all without revealing presence.
• There is little “momentum” between stages of a cyber-attack since the attacker has to reorient themselves, and this incentivizes early intrusion.
• Options for persistence within networks incentivize early intrusion.
• Without moving early, states miss-out on developing economies of scale in capability from re-using code and knowledge
States that desire options purely to defend themselves also have the incentive to intrude early as the defensive process of preparation, detection, data collection, analysis, containment, and decontamination benefit from making intrusions.
• Intruding to learn the strategic intentions of potential intruders
• Intruding to learn potential intruder’s means of intrusion and to develop countermeasures
• Intruding to find out if intrusions have already been made and to discover operations in progress
Cyber intrusions for intelligence gathering will be perceived as more threatening than past intelligence operations
• They provide means of deliberate targeted attacks as well as accidental disruption, since breaking a network can be easier than espionage
• They provide a beachhead for future operations and intrusions
• They change the conditions of conflict and competition via:
• Enabling joint operations with cyber effects
• Providing insight into the defender’s intentions, capabilities, and decision-making processes
• Disclosing the defender’s capabilities in advance
• Enabling economic espionage
• They provide counterintelligence challenges via:
• Intruding into intelligence agency networks to learn about their internal workings, operations, capabilities, communications, methods, and subject knowledge
• Expanding operational opportunities to spread through other intelligence networks
• Undermining intelligence efficiency with fears and paranoia

The traditional mitigations to the security dilemma are less effective for the cybersecurity dilemma

• Cyberspace is perceived as an offense dominant domain of conflict, as it presents large attack surfaces that are difficult to defend since cyberspace is highly interconnected, allows nearly instant intrusion, lacks many parallels to natural “defensive geography”
• The differentiation between offensive and defensive measures and capabilities in cyberspace is not easy for cost-effective means of cyber defense
• Successfully defending and monitoring entire networks and patching all vulnerabilities is prohibitively expensive, so means of defense intrusions into adversary networks to learn about intentions and capabilities are inevitable
• Defensive intrusions can provide means for offensive operations in the future
• It is difficult for states to send credible signals of restraint since the covert nature of cyber operations can conceal their scale and threat
• Resolving uncertainty about motives via communication is more difficult with respect to cyber operations and cyber defenses since states have the incentive to keep secret their capabilities in order not to lose them.

The cybersecurity dilemma is also more complex to solve than the normal security dilemma since cyber capabilities are harder to assess than military power and status quo behaviors are likely to change as norms for mutually acceptable behavior have yet to be ironed out.
Counter-arguments:

In response to these arguments establishing the “cybersecurity dilemma” as a serious problem in international relations, there are a few counter-arguments which the book seeks to address:

• That attribution in cyberspace is impossible…
• Which often is not true for well-resourced actors and even when true is likely to lead defensive forensic intrusions into suspect networks
• That cyber threats are not existential…
• But threats need not be existential to merit or incentivize responses
• That cyber capabilities are so unevenly distributed that detecting a network intrusion does little to impact a state’s overall fears…
• However, states with more symmetric capabilities will still face such dilemmas, and wealthy states face asymmetric threats that still prompt defensive intrusions

Ways to partially mitigate the cybersecurity dilemma:

• Increasing short-term stability with baseline cyber defenses to remove vulnerabilities and increase the odds of purging intruders from networks
• Building trust between states by improving bilateral relationships
• Minimizing the risk of misinterpretation of intentions via unilateral actions to credibly demonstrate commitments to stability (E.g. disclosing zero-day vulnerabilities so they can be patched)
• Establishing and communicating a status quo posture for responding to cyber intrusions

Discuss

### Causal Abstraction Toy Model: Medical Sensor

12 декабря, 2019 - 00:12
Published on December 11, 2019 9:12 PM UTC

Note that, conceptually, there are two main sources of error in the final estimate Y:

• IID measurement noise in the M’s
• Noise in Y from the starting/stopping procedure

… so the node Y is not fully deterministic. The joint distribution for the whole system is given by

P[X,M1...Mn,Y]=P[X](∏iP[Mi|X])P[Y|1n∑iMi]

Since all the measurements are to be averaged together anyway, it would be nice if we could just glom them all together and treat them as a single abstract measurement, like this:

Formally, we can do this in two steps:

1. Replace the nodes M1…Mn with a single node Mall=[M1,…,Mn], i.e. a list containing all the measurements. This doesn’t change the substance of the model at all, it just changes what we’re calling a “node”.
2. Replace the node Mall with M=1n∑Mi∈MallMi, the average of the measurements. We no longer worry about the individual measurements at all, and just directly compute the distributions P[M|X] and P[Y|M].

The second step is the interesting one, since it changes the substance of the model.

Main question: under the abstract model, what counterfactual queries remain valid (i.e. match the corresponding concrete queries), and how do they correspond to counterfactuals on the concrete model? What about probabilistic queries, like P[X|Y]?

The concrete model supports three basic counterfactual queries:

• Set the value of X
• Set the value of Mi
• Set the value of Y

… as well as counterfactuals built by combining multiple basic counterfactuals and possibly adding additional computation. In the abstract model:

• Setting abstract X works exactly the same and corresponds directly to the concrete-model counterfactual.
• Although the abstract Y node has different inputs and computation than the concrete Y node, the procedure for setting abstract Y is exactly the same: cut all the incoming arrows and set the value.
• Setting M corresponds to setting all of the concrete Mi at once, and there may be degeneracy: a single counterfactual setting of M may correspond to many possible counterfactual settings of the whole set of measurements {Mi}.

… so counterfactuals on X and Y have a straightforward correspondence, whereas the correspondence between counterfactuals on M and {Mi} is more complicated and potentially underdetermined. But the important point is that any allowable counterfactual setting of M will correspond to at least one possible counterfactual setting of {Mi} - so any counterfactual queries on the abstract model are workable.

(Definitional note: I’m using “correspond” somewhat informally; I generally mean that there’s a mapping from abstract nodes to concrete node sets such that queries on the abstract model produce the same answers as queries on the concrete model by replacing each node according to the map.)

Probabilistic queries, i.e. P[X|Y], run into a more severe issue: P[X|M]≠P[X|M1,...,Mn]. In the abstract model, node M retained all information relevant to Y, but not necessarily all information relevant to X. So there’s not a clean correspondence between probabilistic queries in the two models. Also, of course, the abstract model has no notion at all of the individual measurements Mi, so it certainly can’t handle queries like P[X|M1].

Now, in our medical device example, the individual measurements Mi are not directly observed by the end user - they just see Y - so none of this is really a problem. The query P[X|M1,...,Mn] will never need to be run anyway. That said, a small adjustment to the abstract model does allow us to handle that query.

Natural Abstraction for the Medical Sensor

Let’s modify our abstract model from the previous section so that P[X|M]=P[X|M1,...,Mn]. Rather than just keeping the information relevant to Y, our M node will also need to keep information relevant to X. (The next three paragraphs briefly explain how to do this, but can be skipped if you're not interested in the details.)

By the minimal map theorems, all the information in {Mi} which is relevant to X is contained in the distribution P[X|{Mi}]. So we could just declare that node M is the tuple (1n∑iMi,(x→P[X=x|{Mi}])), where the second item is the full distribution of X given {Mi} (expressed as a function). But notation gets confusing when we carry around distributions as random variables in their own right, so instead we’ll simplify things a bit by assuming the measurements follow a maximum entropy distribution - just remember that this simplification is a convenience, not a necessity.

We still need to keep all the information in {Mi} which is relevant to X, which means we need to keep all the information to compute P[X|{Mi}]. From the DAG structure, we know that P[X|{Mi}]=1ZP[X]∏iP[Mi|X], where Z is a normalizer. P[X] is part of the model, so the only information we need from {Mi} to compute P[X|{Mi}] is the product ∏iP[Mi|X]. If we assume the measurements follow a maxentropic distribution (for simplicity), then ∏iP[Mi|X]∝e&#x3BB;T∑if(Mi), for some vector &#x3BB; and vector-valued function f (both specified by the model). Thus, all we need to keep around to compute P[X] is ∑if(Mi) - the sufficient statistic.

Main point: the node M consists of the pair (1n∑iMi,∑if(Mi)). If we want to simplify even further, we can just declare that f0 is the identity function (possibly with &#x3BB;0=0), and then node M is just ∑if(Mi), assuming the number n of measurements is fixed.

First and foremost, our abstract model now supports all probabilistic queries: P[X|Y], P[X|M], P[M|Y], P[Y|X], etc, will all return the same values as the corresponding queries on the concrete model (with M corresponding to {Mi}). The same counterfactuals remain valid with the same correspondences as before, and the counterfactually-modified abstract models will also support the additional probabilistic queries.

We can even add in one extra feature:

Huh? What’s going on here?

Remember, M contains all of the information from {Mi} which is relevant to X or Y. That means {Mi} is conditionally independent of both X and Y, given M (this is a standard result in information theory). So we can add {Mi} into the DAG as a child of M, resulting in the overall distribution

P[X,Y,M,Mi]=P[X]P[M|X]P[Y|M]P[{Mi}|M]

Since {Mi} is just a child node dangling off the side, any probabilistic queries not involving any Mi will just automatically ignore it. Any probabilistic queries which do involve any Mi will incorporate relevant information from X and Y via M.

Counterfactual settings of X, Y, and M still work just like before, and we can generally run probabilistic queries involving the Mi on the counterfactually-modified DAGs. Cutting the X→M arrow still corresponds to cutting all the X→Mi arrows in the concrete model. The addition of {Mi} to the model even lets us calculate which {Mi} are compatible with a particular counterfactual setting of M, although I don’t (yet) know of any useful interpretation to attribute to the distribution P[{Mi}|M] in that case.

We still can’t directly translate counterfactuals from the concrete model to the abstract model - e.g. a counterfactual setting of M1 in the concrete model does not easily correspond to anything in the abstract model. We also can’t directly run counterfactuals on {Mi} in the abstract model; we have to run them on M instead. But if a counterfactual modification is made elsewhere in the DAG, the probabilistic queries of {Mi} within the counterfactual model will work.

That brings us to the most important property of this abstraction, and the real reason I call it “natural”: what if this is all just a sub-component of a larger model?

Here’s the beauty of it: everything still works. All probabilistic queries are still supported, all of the new counterfactuals are supported. And all we had to account for was the local effects of our abstraction - i.e. M had to contain all the information relevant to X and Y. (In general, an abstracted node needs to keep information relevant to its Markov blanket.) Any information relevant to anything else in the DAG is mediated by X and/or Y, so all of our transformations from earlier still maintain invariance of the relevant queries, and we’re good.

By contrast, our original abstraction - in which we kept the information relevant to Y but didn’t worry about X - would mess up any queries involving the information contained in {Mi} relevant to X. That includes P[A|{Mi}], P[B|{Mi}], etc. To compute those correctly, we would have had to fall back on the concrete model, and wouldn’t be able to leverage the abstract model at all. But in the natural abstraction, where M contains all information relevant to X or Y, we can just compute all those queries directly in the abstract model - while still gaining the efficiency benefits of abstraction when possible.

Discuss

### [Link] (EA Podcast) Social Status: The Key to the Matrix Part III

11 декабря, 2019 - 23:59
Published on December 11, 2019 8:59 PM UTC

A new episode of Global Optimum has been released! Global Optimum is a podcast aimed at making altruists more effective. This episode is about social status, hypocrisy, signaling, hormones, and politics.

This episode features:

-Why does men’s testosterone go down when they fall in love?

-Does “power posing” have any psychological effects?

-What is “humblebragging” and why does it pervade social media?

-Is our preference for democracy really a preference for high status?

-What is self-esteem?

-How to increase self-esteem (the answer is disappointing)

-How to act high status (the answer is not disappointing)

Full transcript

The podcast is available on all podcast apps.

Listen here

Discuss

### Generalizing Experimental Results by Leveraging Knowledge of Mechanisms

11 декабря, 2019 - 23:50
Published on December 11, 2019 8:39 PM UTC

In a recent post (and papers), Anders Huitfeldt and co-authors have discussed ways of achieving external validity in the presence of “effect heterogeneity.” These results are not immediately inferable using a standard (non-parametric) selection diagram, which has led them to conclude that selection diagrams may not be helpful for  "thinking more closely about effect heterogeneity" and, thus, might be "throwing the baby out with the bathwater."

Taking a closer look at the analysis of Anders and co-authors, and using their very same examples, we came to quite different conclusions. In those cases, transportability is not immediately inferable in a fully nonparametric structural model for a simple reason: it relies on functional constraints on the structural equation of the outcome. Once these constraints are properly incorporated in the analysis, all results flow naturally from the structural model, and selection diagrams prove to be indispensable for thinking about heterogeneity, for extrapolating results across populations, and for protecting analysts from unwarranted generalizations.  See details in the note we post here for discussion.

Discuss

### ToL: Methods and Success

11 декабря, 2019 - 00:18
Published on December 10, 2019 9:17 PM UTC

One of the whole points of this project is to create a clear description of various forms of "success", and to be able to make claims about what is the highest form of success one can hope for given the problem that is being faced. The ultimate point of this is to have a useful frame for justifying the use of different methods. Now I'll introduce the gist of our formalization of methods so that we can get back to the good stuff.

Often I might use notation like M(±A|E) to highlight "This method is responding to a yes or no question A given evidence E. This is mostly useful to be able to talk about certain relations between the question being asked, and your answers.

Gettier Problem

We are going to look at methods that respond with an articulation of an answer. So M(A|E)=A′ st A′⊆A. There's an interesting reason this matters, and part of it has to do with the Gettier Problem.

The Gettier problem has to do with believing the right thing for the wrong reasons. Consider this example.

There's three possible worlds, boxes are possible info states, and the two Hypothesis A and B are outlined. We haven't given a formal notion of what it means to be Occam / "to act in accord with simplicity". But pretend we have. An occam method would say:

M(E1)=A, M(E2)=B and M(E3)=A. In a bit, we're going to make a big deal about the criterion of progressive learning (never drop the truth once you have it). The method I just outline drops the truth in this problem. Suppose w3 is the true world. In E1 is says A, which is true, but then we drop the truth and say B in E2, only to return to it later. Can you see why this is sorta a gettier problem? In E1 we proclaim "A!" but we do it for inductive reasons. It's the simplest hypothesis right now. So it is true that A, but we don't have a super sure reason for saying it. That means that when we get more info, E2, we drop the truth, because our previous "bad reason" for saying A has been disconfirmed.

Our way around these sorts of gettier problems is to not restrict the method to only giving a yes or no answer. That's what an articulation is. A method that gives an articulation would look like M(A|E1)=w1. This makes a lot of sense. The reason you're saying A when you're in E1 is different from the reason you say it in E3. Letting methods give articulations instead of flat yay or nay let's you not loose that information.

Success, Convergences, and Verification

Convergence to an articulation

M converges to an articulation of A in w↔(∃E∈I(w))(∀F∈I(w|E)M(±A|F)⊆A)

Plain English: A method converges to a hypothesis in a given world iff that world has some information state such that no matter what further info you get, your method will stick to it's guns and give an articulation of A

Convergence to a true articulation

M converges to a true articulation of A in w↔(∃E∈I(w))(∀F∈I(w|E))w∈M(±A|F)⊂M(±A|E)⊂A

Plain English: Same as converging to an articulation of A, with the added stipulation that your articulations must include the the world w

Verification in the Limit

M verifies A in the limit in w↔M converges to a true articulation of A if w∈A and M does not converge to a true articulation of A if W∉A

Strong Verification in the Limit

M verifies A in the limit in w↔M converges to a true articulation of A if w∈A and M does not converge to an articulation of A if W∉A

Difference between strong and normal verification:

Consider someone pondering if the true form of the law is a polynomial degree, and what degree it it. This question can be verified in the limit, but it can't be strongly verified. Forever shifting the polynomial degree that you think the law is counts as converging to an articulation of "The true law is polynomial". To strongly verify, at some point your method would have to say "It's not polynomial!" But if you had to keep inter-spacing those in between your other guesses, you don't get to converge at all.

Retractions

Definition: a pair (E,F) st F⊆E∧M(F)⊈M(E) is called a retraction pair.

Here's a picture.

Basically a retraction is any time you get more information and say something that isn't strictly a refinement of your earlier hypothesis. Retractions are really important because they are going to be a key measure of success, one which we connect to various topological properties of a question.

Some brief philosophical motivation for caring about retractions: At first glance, minimizing retractions sounds like being closed minded, and that sounds like a bad quality to have. Luckily, retractions aren't the only thing we're paying attention to when we talk about success. Often we'll talk about converging to the truth while also minimizing retractions. The closed-minded kersmudgeon who sticks to their guns forever doesn't even converge to the truth in most scenarios, and is thus not appealing to us. One way to think about minimizing retractions to "getting to the truth with minimum fuss". It's like missile pursuit.

It's totally expected that for most scientific problems, you're going to have to dodge and weave. But the more of a sequitous path you take in pursuing the truth, the less it feels like it's even right to call what you are doing "pursuit". Converging to the truth while minimizing retractions is like pursuing a target with minimal waste.

A retraction is is a sequence of of info states E0⊇F0 ... En−1⊇Fn−1 such that each pair (Ei,Fi) is a retraction. We'd call this a retraction sequence of length n.

N-verify

Now to the most important definition.

M n-verifies A in w⟺M verifies A in the limit in w and the longest possible retraction chain for M in w is of length ≤n

This concept is about to become very important. A sneak peak at the rest: we have some notion of different types of success you could achieve on a problem. You can verify, refute, or decide a question with 0-&#x3C9; retraction. Next we're going to hop back to topology and construct a topological notion of complexity, one that allows us to make climbs like

A is n-topologically complex ⟺ there exists a method M such that M n-verifies A

If we could do that, then we'd have a way to talk about scientific problems in terms of there complexity, and have a strong way that caches out. For a given problem, you might be able to prove upper or lower bounds on the topological complexity, and thus be able to re-calibrate expectations about what sort of success you can expect from your methods. You might be able to show that a given methods achieves the best possible success, given the topological complexity of the problem. That would be pretty dope. Let's get to it.

(note: So far, for every definition of verification we have given, you can create an analogous definition for refutability and decidability)

Discuss

### ToL: This ONE WEIRD Trick to make you a GENIUS at Topology!

11 декабря, 2019 - 00:02
Published on December 10, 2019 9:02 PM UTC

Time to introduce some new Topological terms. We're going to create some good intuitions around the concepts of interior, exterior, boundary, closure, and frontier. These are all operators in the sense that if you have a set .mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} S, then Int(S) is me using the $$int$$ operator to create a new set that we call "the interior of S". Ext, Bndr, Cl, and Frnt are the shorthand I will use for these operators.

Before talking about these operators in a topological sense, I want to talk about them in a metric space sense. A metric space is just some mathematical space where you have a way to specify the distance between any two points, according to a specific definition of distance. In the real line, the distance between any two numbers can just be the absolute value of their difference. In n-dimensional euclidean space, distance is given by the n-dimensional version of the Pythagorean Theorem. I want to start talking about interiors and boundaries and such from a metric point of view in order to contrast the way it's different from the topological view. I found that when I was trying to wrap my head around these concepts, I was implicitly assuming a metric space world view, because literally every math space I'd interacted with up to that point was a metric space.

The squiggly loop is our set S. In a metric space, a point is in the interior of a set if you can "draw a circle" around it, such that the circle only contains other points that are in S (formally, you talk about "balls" instead of circles. An r-ball around x is the set of all points z st d(z,x)>r). You can clearly see that I can draw a circle around x, where the circle only contains points in S, so x is in the interior of S. A boundary point like y is a point where no matter how small a circle you draw around it, the circle will contain some points in S, and some points not in S.

Likewise, the exterior of S consists of all points that you can draw a circle around such that the circle only contains points not in S. Here are definitions of our two other operators:

ClS=S∪BndrS

FrntS=BndrS∖S

Metrical to Topological

Now, here's where we shift from the metric perspective to the topological perspective. Let's think back to trying to decide if a point is in the interior of S. Re-frame this task as us trying to take a "measurement" around x. "Can I make a measurement that would include x and not include anything from ¬S?" You can see how this is a more general question. We were asking the same question in the metric space context, it's just that our "measurements" were circles of arbitrarily small radii. If were were looking at a space with a. A general metric space abstracts the idea of measuring with a circle to measuring with a ball of arbitrarily small radius. Topology abstracts one step further and says, "we don't even care about distance, we just want to see if you can make some abstract measurement on space that would show x to be surrounded by S."

So what are the "measurements" on a topological space? Its open sets! Remember, a topological space is a set accompanied by a set of things called "open sets" which are subsets of the original set, subject to various axioms. For us, using possible world semantics and the verifiability-topology, we can think of the opens sets (which are all the verifiable propositions) as "measurements" you could take. How does this translate for our topological operators? x is in the topological interior of S if there exists an open set (verifiable proposition) that includes x, and all other members of that open set are members of S. In math, x∈intS⟺∃O∈&#x3C4;:x∈O∧O⊆S

For the rest of this sequence, I'm often going to talk in terms of measurements instead of talking about open sets in the topology. Just know that if you ever get confused, all my statements about measurements should cached out as some statements about open sets, which cache out as statements about the information basis.

At this point, you could re-examine the definitions of the topological operators, swapping out notions of drawing circles with the existance of measurements. Or.... you could check out this one WEIRD PICTURE that will make you A GENIUS at topology!

(Note: I used an upside down "P" because I was going for a mirror symmetry aesthetic, but it didn't work. Just consider the upside down "P" to be Pc or ¬P (they're the same in possible world semantics, remember?). I already made all the images and don't want to change them)

For motivating this picture, consider the open sets of our topology to correspond to rectangles that don't get small enough to cleanly fit into the squiggly boundary and only cover P or Pc.

If you want you can just stare at this picture until you become enlightened. You can also keep reading as I walk through examples.

Decidability

If BndrP=∅ then your problem is decidable. The only possible worlds are ones where you can cleanly measure whether P or ¬P is true.

Verifiable

If Frnt¬P=∅ then P is verifiable: if P is true, it is true in a way that lets us get a clean measurement showing it's true. If it's not true, maybe we get a clean measurement of ¬P, maybe we don't. Note that a problem that is decidable is also verifiable.

Refutable

If FrntP=∅ then P is refutable: if P is false, you can get a clean measurement showing it's false. If it's true, maybe you get a clean measurement of P, maybe you don't.

These pictures help a lot with being able to see what problem statements are duals of each other, and also for translating problem statements into topological statements. See if you can match these problem statements to the corresponding topological ones:

1. P is "strictly" verifiable (verifiable and not decidable)
2. ¬P poses the problem of metaphysics for P
3. P is "strictly" refutable (refutable and not decidable)
4. ¬P poses the problem of induction for P
5. ¬P is "strictly" verifiable
6. "You're fucked"
7. P poses the problem of metaphysics for ¬P

There's lots of other fun exercises you can do to milk intuition from this image. Feel free to play around with it as much as you want. It will be helpful when thinking about and translating between topological ideas in the future.

.

.

.

.

.

.

1. Frnt¬P=ExtP=∅
2. ExtP=∅
3. IntP=FrntP=∅
4. IntP=∅
5. IntP=FrntP=∅
6. IntP=ExtP=∅
7. IntP=∅

Discuss

### What's an important idea you haven't had time to argue for yet?

10 декабря, 2019 - 23:36
Published on December 10, 2019 8:36 PM UTC

Of course, I'm not expecting you to support the idea in the answers, but simply mentioning its conclusion:)

Discuss

### ToL: The Topological Connection

10 декабря, 2019 - 23:29
Published on December 10, 2019 8:29 PM UTC

Now that we've got the basic formalism, it's time to go from our intuitive notion of verifiable to our formal notion of verifiable.

This basically lines up. Something is verifiable if no matter how it's true, there exists some information that would tell you it's true..

Now, a series of leading questions with a fun surprise at the end:

Question 1: Can you verify the contradiction?

Yes! There is no world in which contradiction is true, and with verifiability we only care about what we can do in worlds where the proposition is true.

Question 2: Can you verify the tautology?

Yes! This follows from the first axiom of our info basis. For every world, you get at least one info state. An info state is a subset of P(W), so no matter what the info state we have, it's a subset of W

Question 3: Can you verify the arbitrary finite union of verifiable propositions?

Yes! Takes a little more thought.

Question 4: Can you verify the intersection of any finite number of verifiable propositions?

Okay, get ready for this, here's The Sixth Sense "He was dead the whole time!" plot twist...

(clipped from wikipedia)

... crazy right?

If we take W to be X, and &#x3C4; to be "The set of all verifiable propositions in W", then what we just proved matches up exactly with this definition. So we get to claim that verifiable propositions in W are open sets, and that they form a topology on our set of possible worlds.

What does this mean? From a practical stance, it means that we can now use all of topology, a rich and developed branch of math, to think and talk about verifiability. For any theorem that proves something about opens sets, we can make the same conclusion about verifiable propositions. Before we get into any of that, let's expand our mapping from verifiability concepts to topological concepts.

Definition: A is Closed⟺Ac is open

Okay, so we know that open is verifiable, let's see if we can connect an intuitive notion to closed. Let's look at the Halting Problem. In CS, you'd say the halting problem is semi-decidable. If it's going to halt, you'll see it happen, but if it doesn't you'll never know, maybe it loops forever. Semi-decidable lines up exactly with our notion of verifiability. So what's the negation of the halting problem?

¬HALTS={(M,x):M doesn't halt on x}

Now we've got a situation where if it's true (M doesn't halt), we can't be guaranteed to know). But if it's false (M does halt) we will find out.

Oh shit, this is refutability! To be open is to be verifiable, and to be closed is to be refutable. You can check for yourself, the negation of any refutable proposition is verifiable, and vice verse. Just like how the compliment of any open set is closed, and vice versa.

Discuss

### ToL: Introduction

10 декабря, 2019 - 23:19
Published on December 10, 2019 8:19 PM UTC

(These are the touched up notes from a class I took with CMU's Kevin Kelly this past semester on the Topology of Learning)

Feasibility Contextualism

Despite that, it seems like (at least in terms of watching how people act in the world) everyone gets that you need some level of pragmatism. "Well I've gotta do something, and this seems like the best idea, so I'm going to do it instead of doing nothing." No one is so skeptical of knowledge that they have stayed immobile until they starved to death [citation needed].

What Kelly aims to do is create a rigorous formalism for the sort of pragmatic attitude that people take all the time. It's very much inspired by how computer scientists do things. If someone proves that a problem can't be done any faster than quadratic time, and you figure out a quadratic time algorithm, you're happy. You don't refuse to use any algorithm that doesn't run in constant time.

In a sentence, this course was about how really cool formalism to talk about "How hard is a given scientific problem" and how that effects "the best possible performance you can get given the hardness of the problem".

Sneak Peak: Induction and Metaphysics

Over the ages people have postulated what qualities scientific hypothesis should have. The logical positivists asserted that only verifiable propositions should be the domain of science (if it's true, you can do some test to demonstrate it's true). Popper wanted hypothesis to be falsifiable (if it's false, you can do some test to demonstrate it's false). Verification and falsifiability have an important connection to two other notions that philosophers of science often talk about, the problem of induction and the problem of metaphysics.

You face the problem of induction if it's the case that even if you hypothesis is true, you'll never know for sure (will the sun rise tomorrow? You can never rule out that it just won't at some point).

You face the problem of metaphysics if it's the case that even if your hypothesis is false, you'll never get definitive evidence that it's false ("there's a teacup somewhere in the infinite expanse of space!")

Turns out almost all questions that have been the subject of actual science are neither verifiable or falsifiable, and you're going up against induction or metaphysics with most questions as well. Uh Oh. Looks like those normative ideas on what science should be rule out most of what science is. Oops.

In light of our new tack on things, this is sorta like if the only complexity classes people had were linear and quadratic, and thought that the only problems that should be in the domain of computer science should be ones that are solvable in linear or quadratic time. Fix: make a richer complexity hierarchy in which to locate problems, see what the complexity says about possible performance.

Discuss

### ToL: Foundations

10 декабря, 2019 - 23:15
Published on December 10, 2019 8:15 PM UTC

(this post is part of my touched up notes for a class I had on philosophy of science. Only partially optimized for making sense)

Possible World Semantics

Now we consider some true or false proposition A concerning W. A could be "There are more than 15 people in North America". The key idea in possible world semantics is that every proposition A is represented by the set of all worlds where A is true.

Def: Proposition A={w∈W:A is true in w}

We'll still refer to propositions by their English sentence description, but when we start doing math with them you should think of them as sets of worlds. Here are some consequences of defining logical propositions as such:

1. A∨B=A∪B
2. A∧B=A∩B
3. ¬A=Ac
4. A→B=A⊆B
5. ⊤=W
6. ⊥=∅

Convince yourself that the above are true. If you're wondering, "Hmmm, the logic of set containment and classical logic seem eerily similar" you're right, and might want to ponder if that has anything to do with how we decided what the rules of classical logic should be in the first place.

Information Basis

Now that we have our world, the next thing we want is our information basis I. I is made up of information states, and each information state is a proposition. This means that they follow all the same union and intersection rules that other propositions do, and that an info state is a set of possible worlds. However, your info basis can't just be any old set of proposition. We are trying to capture the set of propositions that you could know about the world. Upfront, I want to acknowledge that this "could" might cause some confusion. How do we know what you could or couldn't know? Isn't that what we're trying to figure out? For now, we will resolve that with the following distinction. When we're talking about possible worlds and information states, they are not defined by some requirement that a particular person could access them. Later when we talk about methods, then we'll be talking about, "What conclusions could someone with XYZ method reach?"

The information basis, just like the possible worlds, will be shaped by how we construct the setup to our inquiry. Normally, what the information basis looks like will be a direct result of what "measurement" tools are being used to do inquiry. Separate from what the basis looks like in any given construction, below are some basic axioms that we always have our info basis abide by.

1. ∀w∈W:I(w)≠∅
2. ∀w∈W:∀A,B∈I(w):∃C∈I(w):C=A∩B
3. I is countable

The first axiom just states, "No matter what world you're in, there's something you can know, even if it's just the tautology". This is mostly a bookkeeping axiom and doesn't have profound philosophical consequences.

The third axiom has some intuitive appeal (it's hard to imagine finite beings interacting with uncountable entities) but again is mostly a bookkeeping axiom to make some proofs slicker down the road.

The second axiom is the interesting one. It plain English it says "For any two info states you could witness, there is another info state that includes the information of both." This can be thought of "additivity of evidence". If there are two propositions A and B, two possible things you could know, then it is "possible" to know both of them. There won't be a weird branching of your experiments where if you see A you'll never be able to see B.

Okay, that's a bulk of the setup to the relevant syntax, and it probably wasn't very insightful or meaningful to you. Let's hop into some examples and see what it looks like to model problems in this framework.

Example: Hume's Black Box

I'm going to model a simple inductive problem. Let's say everyday you wake up and check whether or not aliens have made contact with earth. Everyday you put a "0" up on the wall if they haven't, and a "1" if they have.

In this setup, a possible world is any given infinite sequence of 1's and 0's. Something like:

wmine={00000000000000000000000000...}

This makes the set of all possible worlds the set of all possible infinite binary strings.

W=2&#x3C9;

(notation explanation: XY​ is common notation for "all function from Y​ into X​". So 2&#x3C9; is "all function from the naturals to 2 (which in many set theory construction, 2 is defined to be the set {0,1}​)". A function from the naturals to the set {0,1}​ defines an infinite binary string)

Onto our info states. Since we are observing this infinite sequence day by day, when can only ever have seen a finite amount of it. So we probably want an info state to be something like e∈2<&#x3C9; where ​2<&#x3C9; is "all finite binary strings". But remember, an info state has to be a proposition, and a proposition is a set of possible worlds. No world is represented by a finite binary string. So we do the following:

[e]={w∈2&#x3C9;:e∈w}

I={[e]:e∈2<&#x3C9;}

And boom, we've got our info basis.

Here's what this looks like as a picture:

The circles represent information states. There is an information state that confirms "Bread fails to nourish at time t = 3", and there are information states like "either bread nourishes or it fails to nourish at t > n", but there is no information state that uniquely picks out the world where bread nourishes, which is why this is an induction scenario.

Example: Function Learning

There exists some function on the real numbers, and we are trying to figure out what sort of function it is. We get to investigate the function by getting arbitrarily small rectangle measurements of it.

W={f:R→R}

[(x1,x2),(y1,y2)]={f∈RR:∃x st x1≤x≤x2∧y1≤f(x)≤y2}

I={[(x1,x2),(y1,y2)]:x1<x2∧y1<y2}

The motivation for the rectangular measurement is to account for measurement error. Imagine there is some natural law, and investigating the function is us setting one variable and seeing how another variable changes. There's there small uncertainty, we never actually check the function at a point and get info like "f(15.4) = -37". You put in an approximate input and get an approximate answer. You can refine the approximation as much as you want and get the error smaller and smaller, but there is never zero error.

Discuss

### Applications of Economic Models to Physiology?

10 декабря, 2019 - 21:09
Published on December 10, 2019 6:09 PM UTC

Applying economic models to physiology seems really obvious. For instance:

• Surely the body uses price signals to match production to consumption of various metabolites. Insulin as a price signal for glucose is one example.
• Presumably such price signals coordinate between spatially-separated organs with specialized roles in various physiological "supply chains". That should lead to general equilibrium models, and questions of convexity and stability.
• Can we back out an implied discount rate for the body's long-term energy stores?

Yet when I run a google search for the obvious phrase "econophysiology", I get back five results, most of which appear to be misspellings. (I feel like I ought to write something right now just to call dibs on the name.)

Does anyone know of sources on this sort of thing? Is there a name for it?

Discuss

### Predictive coding = RL + SL + Bayes + MPC

10 декабря, 2019 - 14:45
Published on December 10, 2019 11:45 AM UTC

I was confused and skeptical for quite a while about some aspects of predictive coding—and it's possible I'm still confused—but after reading a number of different perspectives on brain algorithms, the following picture popped into my head and I felt much better:

This is supposed to be a high-level perspective on how the neocortex[1] builds a predictive world-model and uses it to choose appropriate actions. (a) We generate lots of "hypotheses", a.k.a. generative models (also called "patterns" by Kurzweil, or "subagents" by Kaj Sotala[2]) in parallel about what's going on and what's going to happen next, including what I am doing and will do next (i.e., my plans). The hypotheses gain "prominence" by (b) correctly predicting upcoming sensory inputs; (c) correctly predicting other types of input information coming into the neocortex like tiredness, guilt, hunger, warmth, pain, pleasure, and so on; (d) being compatible with other already-prominent hypotheses; (e) predicting that my innate drives will be satisfied—my goals will be fulfilled with minimal effort, I'll be eating soon if I'm hungry, I'll be sleeping soon if I'm tired, I'll avoid pain, and so on. Whatever candidate hypothesis winds up the most "prominent" wins, and determines my beliefs and actions going forward.

Before we get to details, I need to apologize for the picture being misleading:

• First, I drew (b,c,d,e) as happening after (a), but really some of these (especially (d) I think) work by affecting which hypotheses get considered in the first place. (More generally, I do not want to imply that a,b,c,d,e correspond to exactly five distinct neural mechanisms, or anything like that. I'm just going for a functional perspective in this post.)

• Second (and relatedly), I depicted it as if we simply add up points for (b-e), but it's certainly not linear like that. I think at least some of the considerations effectively get vetoes. For example, we don't generally see a situation where (e) is so positive that it simply outvotes (b-d), and thus we spend all day checking our wallet expecting to find it magically filled with crisp 1000 bills. (Much more about wishful thinking below.) • Third, at the bottom I drew one hypothesis being the "winner". Things like action plans and conscious attention do in fact have a winner-take-all dynamic because, for example, we don't want to be sending out muscle commands for both walking and sitting simultaneously.[3] But in general, lower-ranked hypotheses are not thrown out; they linger, with their prominence growing or shrinking as more evidence comes in. Anyway, the picture above tells a nice story: (b) is self-supervised learning[4], i.e. learning from prediction. Process (b) simply votes against hypotheses when they make incorrect predictions. This process is where we get the vast majority of the information content we need to build a good predictive world-model. Note that there doesn't seem to be any strong difference in the brain between (i) actual experiences, (ii) memory recall, and (iii) imagination—process (b) will vote for or against hypotheses when presented with any of those three types of "evidence". (c) is credit assignment, i.e. learning what aspects of the world cause good or bad things to happen to us, so that we can make good decisions. Each hypothesis makes claims about which part of its models are the cause of subcortex-provided informational signals (analogous to "reward" in RL)—information signals that say we're in pain, or eating yummy food, or exhausted, or scared, etc. These claims cash out as predictions that can prove right or wrong, thus either supporting or casting doubt on that hypothesis. Thus our internal models say that "cookies are yummy", corresponding to a prediction that, if we eat one, we'll get a "yummy" signal from some ancient reptilian part of our brain. (d) is Bayesian priors. I doubt we do Bayesian updating in a literal mathematical sense, but we certainly do incorporate prior beliefs into our interpretation of new evidence. I'm claiming that the mechanism for this is "hypotheses gain prominence by being compatible with already-prominent hypotheses". What is an "already-prominent hypothesis"? One that has previously been successful in this same process I'm describing here, especially if in similar contexts, and super-especially if in the immediate past. Such hypotheses function as our priors. And what does it mean for a new hypothesis to be "compatible" with these prior hypotheses? Well, a critical fact about these generative models is that they snap together like Legos, allowing hierarchies, recursion, composition, analogies, casual relationships, and so on. (Thus, I've never seen a rubber wine glass, but I can easily create a mental model of one by gluing together some of my rubber-related generative models with some of my wine-glass-related generative models.) Over time we build up these super-complicated and intricate Rube Goldberg hypotheses, approximately describing our even-more-complicated world. I think a new hypothesis is "compatible" with a prior one when (1) the new hypothesis is almost the same as the prior hypothesis apart from just one or two simple edits, like adding a new bridging connection to a different already-known model; and/or (2) when the new hypothesis doesn't make predictions counter to the prior one, at least not in areas where the prior one is very precise and confident.[5] Something like that anyway, I think... (e) is Model-Predictive Control. If we're hungry, we give extra points to a hypothesis that says we're about to get up and eat a snack, and so on. This works in tandem with credit assignment (process (c)), so if we have a prominent hypothesis that giving speeches will lead to embarrassment, then we will subtract points from a new hypothesis that we will give a speech tomorrow, and we don't need to run the generative model all the way through to the part where we get embarrassed. I like Kaj Sotala's description here: "mental representations...[are] imbued with a context-sensitive affective gloss"—in this case, the mental representation of "I will give a speech" is infused with a negative "will lead to embarrassment" vibe, and hypotheses lose points for containing that vibe. It's context-sensitive because, for example, the "will lead to feeling cold" vibe could be either favorable or unfavorable depending on our current body temperature. Anyway, this framing makes a lot of sense for choosing actions, and amounts to using control theory to satisfy our innate drives. But if we're just passively observing the world, this framework is kinda problematic... (e) is also wishful thinking. Let's say someone gives us an unmarked box with a surprise gift inside. According to the role of (e) in the picture I drew, if we receive the box when we're hungry, we should expect to find food in the box, and if we receive the box when we're in a loud room, we should expect to find earplugs in the box, etc. Well, that's not right. Wishful thinking does exist, but it doesn't seem so inevitable and ubiquitous as to deserve a seat right near the heart of human cognition. Well, one option is to declare that one of the core ideas of Predictive Coding theory—unifying world-modeling and action-selection within the same computational architecture—is baloney. But I don't think that's the right answer. I think a better approach is to posit that (b-d) are actually pretty restrictive in practice, leaving (e) mainly as a comparitively weak force that can be a tiebreaker between equally plausible hypotheses. In other words, passive observers rarely if ever come across multiple equally plausible hypotheses for what's going on and what will happen next; it would require a big coincidence to balance the scales so precisely. But when we make predictions about what we ourselves will do, that aspect of the prediction is a self-fulfilling prophecy, so we routinely have equally plausible hypotheses...and then (e) can step in and break the tie. More general statement of situations where (e) plays a big role: Maybe "self-fulfilling" is not quite the right terminology for when (e) is important; it's more like "(e) is most important in situations where lots of hypotheses are all incompatible, yet where processes (b,c,d) never get evidence to support one hypothesis over the others." So (e) is central in choosing action-selection hypotheses, since these are self-fulfilling, but (e) plays a relatively minor role in passive observation of the world, since there we have (b,c) keeping us anchored to reality (but (e) does play an occasional role on the margins, and we call it "wishful thinking"). (e) is also important because (b,c,d) by themselves leave this whole process highly under-determined: walking in a forest, your brain can build a better predictive model of trees, of clouds, of rocks, or of nothing at all; (e) is a guiding force that, over time, keeps us on track building useful models for our ecological niche. One more example where (e) is important: confabulation, rationalization, etc. Here's an example: I reach out to grab Emma's unattended lollipop because I'm hungry and callous, but then I immediately think of an alternate hypothesis, in which I am taking the lollipop because she probably wants me to have it. The second hypothesis gets extra points from the (e) process, because I have an innate drive to conform to social norms, be well-regarded and well-liked, etc. Thus the second hypothesis beats the truthful hypothesis (that I grabbed the lollipop because I was hungry and callous). Why can't the (b) process detect and destroy this lie? Because all that (b) has to go on is my own memory, and perniciously, the second hypothesis has some influence over how I form the memory of grabbing the lollipop. It has covered its tracks! Sneaky! So I can keep doing this kind of thing for years, and the (b) process will never be able to detect and kill this habit of thought. Thus, rationalization winds up more like action selection, and less like wishful thinking, in that it is pretty much ubiquitous and central to cognition.[6] Side note: Should we lump (d-e) together? When people describe Predictive Coding theory, they tend to lump (d-e) together, to say things like "We have a prior that, when we're hungry, we're going to eat soon." I am proposing that this lumping is not merely bad pedagogy, but is actually conflating together two different things: (d) and (e) are not inextricably unified into a single computational mechanism. (I don't think the previous sentence is obvious, and I'm not super-confident about it.) By the same token, I'm uncomfortable saying that minimizing prediction error is a fundamental operating principle of the brain; I want to say that processes (a-e) are fundamental, and minimizing prediction error is something that arguably happens as an incidental side-effect. Well, that's my story, it seems to basically makes sense, but that could just be my (e) wishful thinking and (e) rationalization talking. :-) 1. The neocortex is 75% of the human brain by weight, and centrally involved in pretty much every aspect of human intelligence (in partnership with the thalamus and hippocampus). More about the neocortex in my previous post ↩︎ 2. Some hypotheses are not at all subagent-y, like the hypothesis "That is a falling ball and it is going to bounce.". Other hypotheses are very subagent-y, like the hypothesis "That is a dangerous hot stove, and I am going to run away from it, and then I will be safe!". ↩︎ 3. I described conscious attention and action plans as "winner-take-all" in the competition among hypotheses, but I think it's somewhat more complicated and subtle than that. I also think that picking a winner is not a separate mechanism from (b,c,d,e), or at least not entirely separate. This is a long story that's outside the scope of this post. ↩︎ 4. I have a brief intro to self-supervised learning at the beginning of Self-Supervised Learning and AGI Safety ↩︎ 5. Note that my picture at the top shows parallel processing of hypotheses, but that's not quite right; in order to see whether two prominent hypotheses are making contradictory predictions, we need to exchange information between them. ↩︎ 6. See The Elephant in the Brain etc. ↩︎ Discuss ### [Review] On the Chatham House Rule (Ben Pace, Dec 2019) 10 декабря, 2019 - 03:24 Published on December 10, 2019 12:24 AM UTC This is a brief review of On the Chatham House Rule by Scott Garrabrant. I tend to be very open about my thoughts and beliefs. However, I naturally am still discrete about a lot of things - things my friends told me privately, personal things about myself, and so on. This has never been a big deal, figuring out the norms around secrecy. For most of my life it's seemed pretty straightforward, and I've not had problems with it. We all have friends who tell us things in private, and we're true to our word. We've all discovered a fact about someone that's maybe a bit embarrassing or personal, where they don't know we know, and so we've e.g. carefully moved a group the conversation around not putting pressure on them to explain why they were unavailable last Tuesday. Discretion is a tool we all use implicitly, and often successfully. And yet, in the last two years, I've had to think very carefully about norms around secrecy, and found increasingly difficult problems that have substantially changed my behaviour. I think the main change is that I much more regularly start interactions with 5-25 minutes of conversation about the expected disclosure norms. And furthermore, I regularly decline information that comes with strong nondisclosure norms attached, after spending a minute talking about the terms and conditions of secrecy. Now, why do I do this? And what has changed? I'm not certain. I think I've not enjoyed the weird politicisation of language that happens behind secrecy, where people use abstract words a lot, and then don't correct you when you misuse them, because they can't tell you the thing they were actually thinking of. I especially don't like talking with people who aren't even telling me that they're hiding secret information that informs their opinions, sometimes this feels outright deceptive. More than either of these though, I don't like being in that position myself. I like to just say my thoughts out loud in conversation. It's very, very limiting to not be allowed to just answer when someone asks me why I believe what I believe. Note that there's a big difference between not letting certain information out, and recomputing how you would reply and what thoughts you would have if you had never heard that information in the first place. Riffing off of meta-honesty: it's very hard to be meta-open about openness norms. "I see, so you don't want to tell me your opinion on <topic> because you have secret information. I think this is mostly pretty bad for your and my ability to talk and coordinate on this topic. Here are <detailed reasons>. Can you tell me why you disagree with those reasons?" "No, because that would be giving away my secret information about <topic>." This is a conversation I've seen happen, where a researcher I respect sat down with a friend of mine to discuss secrecy, and 10 mins into the conversation the researcher realised they weren't able to say their reasons because of secrecy concerns. One of the main points of Bostrom's career is that not all ideas should be let out, certainly not in any order. We must become much wiser as a civilization before the maximally free exploration of ideas is universally safe, and I don't think that many people are wrong to keep information secret. My main concern about secrecy is that people do not put in the work to ensure that the public discourse is maintained once people and organisations go dark. But that's a story for another day. Figuring out how to do secrecy well is important, and hard. Scott's post On The Chatham House Rule takes a fairly common set of explicit secrecy norms, and shows how much more demanding they are than anybody expected, and how lack of clarity around that has lead to their failure to work. It's a helpful example of showing how trying to have such norms is difficult and requires careful thought. I think secrecy is very hard, and we now know that just saying "Chatham House Rules" does not work. When it was first posted I thought it was just an irritating technicality, and didn't think much of the post. But now I see it as part of a broader pattern where secrecy norms have much more wide-ranging implications than one often expects, and require a lot of careful thought to get right. I appreciate having such a clear public example to point to of how simple secrecy norms, that everyone thinks are simple, are not simple at all, are confusing, and break easily. It is nothing like the last word on the subject, but a useful early step in the conversation. Discuss ### [Review] Meta-Honesty (Ben Pace, Dec 2019) 10 декабря, 2019 - 03:14 Published on December 10, 2019 12:14 AM UTC I read and really liked Meta-Honesty: Firming Up Honesty Around Its Edge Cases, by Eliezer Yudkowsky. Here I’ve tried to write down how it fits into my thinking. • In the first section of this post I write down my own thoughts on honesty. In particular I list many edge cases that I have difficulty with. • In the second section, I explain what Eliezer’s post recommends when dealing with edge cases. • In the third section, I talk about whether we can build common knowledge of Eliezer's new rule, and what effects that might have. Finally, I close with a summary of everything discussed. Honesty and its Edge Cases While there aren't very many people who think lying is a virtue, there are lots of people who don’t mind fudging the truth, especially when it seems useful/harmless. I am not one of those people. I put in a lot of work to make sure things I say are true, that you can trust them, and don't have to worry about me saying things just because it's convenient or I don't care about you. I will still say true things. But this is not a clear line to walk. There are lots of situations where it’s hard to keep up to an ideal standard of honesty, where people get false conclusions from what I say. Context Failures If I tell a joke, and someone overhears it but doesn't catch the punchline, and thinks I'm just telling a true story, does that mean I've failed to ensure people can trust what I say? What if I make a taboo-breaking (e.g. sexist) joke to a friend, and someone else overhears? I'd only make that joke in a setting where there was no doubt it was a joke. I wouldn't make the joke in public. But they've overheard me, and now I've managed to send inaccurate information out about what I consider to be socially acceptable. In general, group contexts can be very hard to compute the effects of speech for (i.e. having to think about what others expect you to think is common knowledge) and what you say expecting several people to hear one thing, sounds very different to another person, because of the sorts of things they think it's acceptable to omit, and just because someone cares about honesty does not always mean that I expect in groups of people they will not end up sending false signals to some people listening. What if I'm thinking about a possible project at work, and we're talking really excitedly about the idea? We keep discussing the details of what we're going to do, sharing our impressions, not our reflective all-things-considered views. There's a chance that a person walks away thinking that if you ask me to assign a probability distribution over what project I'll work on next, that I'd assign 80% to that one. But in truth, the connection between the immediate impressions I share and my reflective probabilities about predicting my own actions is very non-obvious and difficult to trace, and I honestly am still withholding judgement about what I'll do next. Am I responsible for that false impression? Do I need to pepper qualifications throughout everything I say? Do I always need to say "And of course my all-things-considered probability of us doing this is only 10%" at the end of every conversation? All the above cases are about miscommunication that I would fix if I knew they'd happened. But there are cases where that isn't what's going on. There are sometimes cases where I choose to say something I know is inaccurate or deceptive. Here are three examples. Threats from powerful institutions However, I'm still talking around outright deception. I feel like once again biting off an edge-case and talking about things like when the airport person asks you to lie. "Have you let these bags out of your sight at any point since packing them?" to which the only answer that gets you on the plane is "No". I'm not allowed to say "I understand legally it's required that, as evaluated by a hypothetical future court of law, your cognitive state must be justified in believing me so I have to do the work to deceive you. And so I will tell you 'no'." Or say anything like that, which creates honest communication and plausible deniability in a court of law. If I want to travel, I have no choice but to actively do a tiny bit of deception here. And that's a hat I can wear, with schoolteachers and bureaucracies and so on. I don't like that hat, and wear it less and less as I've become an adult. Again, I'm trying to think of real lies I've told in my life. Okay, I've definitely told many active lies to people at institutions that I think have got me in a corner with no way out, who've held my life in their fickle hands, and I do that very dirty-feeling where a bit of acting is involved. Governments, universities, and the like. They mostly feel similar to the airport situation, although sometimes there's a bit of acting involved or longer-term lie upkeep. It's hard to describe a principled way of separating the two. That said I try to be honest about that fact, and I think I'd even be happy to tell some of the people at those institutions about it, especially the smarter ones in the computer science department who can do that decoupling norms thing. As long as they understood that I was allowed to decline them asking about specific things we'd discussed together. Fighting politicised narratives Changing tack a little, one of the main kinds of honesty that's difficult for me is figuring out when to actively speak up. When a powerful actor / group of actors is sustaining a narrative that is helpful to them, there are many choices. One can be silent. One can speak up against the narrative. One can build common knowledge amongst a small group of friends and try to grow it over time. A weirder and harder option is to figure out how you'd talk about the subject if there weren't such a powerful actor threatening punishment, and then put in effort to talk about it exactly like that, including in casual conversation and public settings. I think that last one can be especially valuable for group epistemology, but can also be very computationally expensive, to figure out where my mind would naturally go if certain threats weren't salient. I don't think I currently have any simple, explicit rules here, especially not those that are very context independent (there are several topics I feel silenced on, and I could talk about what heuristics I use in those areas, but I don't want to because I don't feel like casually naming the topics here). Personal Compute Limitations Another of the main one that comes up for me is around personal compute limitations. I just do not have the time and effort to give someone a long explanation of the details of a thing, and they're asking me for a quick question with a quick answer, then it’s often easier to give a short answer that helps them locally but has some false implications. Like, if we're leaving the house and someone asks "Is it going to rain?" I try to never say "Yes" when I instead mean "Well, probably not, but the expected utility of taking the umbrella is still worth it given how irritable we'd be if a heavy downpour hit us, and yes, I do think the probability of that is sufficiently high as to be worth it." If I think quickly enough I manage to change the frame. They will say "Is it going to rain?" and I will respond "You should take the umbrella." But I'm not perfect, and sometimes I have a billion things going on and I'm cognitively exhausted and I'm irritated by the person and so on, I... well, sometimes I consider just saying “Yes.” Let me write down something like an internal dialogue I might have in those situations. "You’re telling an untruth!" "I agree and it’s unfortunate, but while I do have the cognitive resources to run this 'check for explicit truth of my words' subroutine, I don't have the resources to run the 'figure out how to communicate this info correctly while navigating the social interactions with this particular person' subroutine because that subroutine is so tiring and painful right now, so for now I think I endorse just saying 'yes' and mark this as a situation where Ben isn’t trustworthy." “Really? Is this the line where I want to tell my friends they can’t trust me? ‘Oh, if I’m tired and am finding you a bit irritating then be on guard, because you can no longer trust me to not say things that will mislead you.’ That doesn’t sound like where I want to draw my boundary. So I think we can weather this particular situation and still say true things, don’t you?” “I agree that’s the right thing to do in general, but I’ve been doing it a lot lately, and I’m just tired. I think this situation and is a build-up of a lot of long-term problems with this particular person where I’m not getting some pretty core needs met in my life, and don’t expect to any time in the next 6 months at minimum, which is a sufficiently difficult situation that I think I can’t promise to keep my integrity here, and I should put this on the list of things I explicitly tell my friends and potential future collaborators, so that they know what sorts of deals we can agree to together - what sorts of commitments I can actually make, through exactly what level of tough times.” “I know, and I hear that, but I think it’s good to be able to promise it still in this level of difficult situation, and I honestly think you’ll be okay a year from now, and if you focus on that you’ll have the energy to get through now, even though it is difficult.” “I agree, things will probably get better, and during the rest of my life I’ll appreciate that I stuck through this time without having told even one mistruth. And focusing on that has given me a little energy. Okay, I’ll run the subroutine.” Further Thoughts on Honesty Some people, on hearing the difficult challenges of honesty, would call these problems defeaters, which make 'honesty' meaningless. I would counter that putting more effort into accuracy shrinks the gap between what you say and what people believe you said, so while perfect platonic communication is not possible as we currently understand it, there is a definitely a dimension here you can push on, and this matters. I'm mostly talking here from my inside view of what trying to be honest feels like. Robin Hanson would not at all be convinced that I'm very honest by the above, and would point me to each and every chapter of his book 'The Elephant in the Brain' and describe how I'm clearly being dishonest when it comes to medicine, friendships, education, and so on, and generally argue that I'm not the sort of machine that can be honest. That's an important viewpoint, and clashing it with this one is a good thing to do, but I won't do it in this post, because I'm mostly trying to lay out the particular inside view of honesty I have that Eliezer's post fits into. Meta-Honesty So where does Eliezer's essay fit into this? The essay takes the same starting assumption as I do. Honesty is deadly important and something you work hard at. There are some edge cases though, as above, and Eliezer tries to make a simple rule that captures a lot of them. Eliezer's general approach is not case-based as mine is above, but more law-based, where he looks for general rules, and considers individual cases insofar as they provide him with forced-moves from the rule-based perspective. The things that motivate him aren't individual experiences, but are: • The fact that it’s easier to always say things that are technically true if you’ve got high verbal intelligence, and that this means requiring absolute honesty is unfairly easier for some than for others. • The massive number of counterfactual versions of you who would like you to keep plausible deniability when answering questions like “What did you do last night?”. • The case of hiding Jews in the attic and answering to Nazis who ask if you are hiding Jews. • Robin Hanson's problem of automatic norms, where people judge others for not immediately knowing their own norms. This induces (amongst many things) some self-doubt about whether your norms are as obviously correct as you’ve been thinking. Instead of coming up with bits of advice here and there, he comes up with a new rule. Eliezer's rule is fairly straightforward. He acknowledges that, yes, there are situations where even a very honest person will fail to be honest. In line with what I've said above, there are many edge cases. And so he simply suggests that on top of this, you should be absolutely honest about where you'll likely be honest and dishonest. Here’s how he puts it. Be at least as honest as an unusually honest person. Furthermore, when somebody asks for it and especially when you believe they're asking for it under this code, try to convey to them a frank and accurate picture of the sort of circumstances under which you would lie. Literally never swear by your meta-honesty that you wouldn't lie about a hypothetical situation that you would in fact lie about. In many of the edge cases I listed above, I have genuinely sometimes communicated falsehoods to someone (wittingly and unwittingly). The important thing that Eliezer adds is that if they or others were to ask me in general about such situations, I would tell them, (or at least explicitly remain silent if I felt I had to). "Yes, if I feel backed into a corner by a pathological institution who requires me to say certain things with conviction in order to even get on a plane, I am likely to say those things." "Yes, if I feel I've totally run out of cognitive resources and don’t expect to be able to replenish them in the next couple of months, I may tell you a simple but slightly misleading answer that will lead to the right local action." "Yes, if I believed a friend of mine was being unjustly imprisoned entirely by a corrupt government, I would be likely to act in a way I determine most helpful to my friend being released, and may state outright falsehoods." And so on. Eliezer writes: [Living out meta-honesty in real life means] stopping and asking yourself "Would I be willing to publicly defend this as a situation in which unusually honest people should lie, if somebody posed it as a hypothetical?" I can't require everyone around me to be more honest than I think is possible for myself. Nonetheless, we need to find our way to a strong position where we can discuss the edge cases and get an understanding of how we each draw those boundaries. If we have differences in the boundaries and don’t hold ourselves to this standard, it will be very hard to cooperate and build agreements on trust, not least because we're misleading each other about our thoughts and actions in these areas. So when we're talking more generally about honesty, it's important to be able to needle in on the reasoning we use when making calls in the tough cases, and not pull any clever tricks on the meta level. I've pointed to some of my own above, and I expect this will lead even some of my close friends to say "Huh, I guess I do things a bit differently. Let's talk about it some, so that we don't mislead each other or fall importantly short of each other's standards." I don’t know whether there is a single standard of honesty that I can apply to all people. Some of us are stronger than others, and will be able to uphold honesty under harsher situations than others. I know people stronger than me, I know people weaker than me. But we need to be open about that, and talk about it. And when discussing that, we should be absolutely honest. Virtues and Rules It’s good to be an honest person. I commonly take the virtue ethics perspective on honesty, and as a virtue, I think it’s one of the most important ones to live up to, up there with courage and curiosity and maybe one or two others. I think that Eliezer’s post is attempting to be a straightforward, somewhat theoretical addition to that vision - or rather, it’s an attempt to make a reflectively consistent update to that vision. I think if I advised someone to be an honest person, and then gave them the further advice in Eliezer’s post, this would seem like a natural extension and not obviously inconsistent. I’m not certain that all of Eliezer’s details are correct, and I look forward to others thinking more about this, but it feels like distinct progress to me. Moving to deontology, where we try to make our morals a little more explicit, and write down rules. As above, there are many edge cases, and the deontology does not weather too close an examination. But I’ve tried at times to live up to the deontological rule of pure honesty, and it’s been intensely refreshing. Starting to say that you’ll do something later, and stopping when you realise you’re making a false promise, and instead stating the true state of affairs (“...I would like to do that for you... but I can’t do it right this minute... and you should actually only assign a 30% probability to me remembering to do it later unprompted. Having realised that, let me get out my phone and set a reminder.”) It’s a great step toward becoming a robust agent Can We Build Common Knowledge Of Meta-Honesty? The question to grapple with here is not whether this analysis of honesty is a valuable step toward understanding the underlying Laws, but is whether in its current form we can build a communal norm around meta-honesty. Thou shalt be meta-honest. Is understanding meta-honesty something we can build common knowledge around? I ask whether we can build common knowledge of meta-honesty, not of ‘meta-honesty’. The substance, not the linguistic name. Eliezer is worried about miscommunication of this idea leading to an erosion of norms. He writes: So there's still a very obvious thing that could go wrong in people's heads, a very obvious way that the notion of "meta-honesty" could blow up, or any other code besides "don't say false things" could blow up. It's why the very first description in the opening paragraphs says "Don't lie when a normal highly honest person wouldn't, and furthermore…" and you should never omit that preamble if you post any discussion of this on your own blog. THIS IS NOT THE IDEA THAT IT'S OKAY TO LIE SO LONG AS YOU ARE HONEST ABOUT WHEN YOU WOULD LIE IF ANYONE ASKS. It's not an escape hatch. If anything, meta-honesty is the idea that you should be careful enough about when you break the rule "Don't lie" that, if somebody else asked the hypothetical question, you would be willing to PUBLICLY DEFEND EVERY ONE OF THOSE EXTRAORDINARY EXCEPTIONS as times when even an unusually honest person should lie. How clear is the post? When trying to offer practical advice in the form of rules, rather than trying to infinitely justify every definition and clause to all possible objections, I tend to think that the best way of doing things is to just instead show them what the thought processes of a person using the rule looks like. Show-don't-tell. This is a skillset that Eliezer has worked harder on than maybe anyone else I know, and I think Eliezer's essay genuinely gets across much of that kind of high-bandwidth information - especially sections 6 and 7, which show the details that sway his internal dialogue in concrete (and sometimes fictional) situations. Er, actually, saying that aloud I'm not so sure[1]. I went into Eliezer's essay several times before I felt I understood it all. Especially the key sections of 6 and 7, with all their extended discussion about glomarizing. He even wrote a small Harry-Dumbledore dialogue to help, and it was still tough. My friends also needed multiple reads. On initially reading the post that I thought it was proposing the norm that you could lie as long as you were open about that fact. The fact that Eliezer stated the opposite in capital letters was important and helpful for me not misunderstanding him, but I still didn’t get what it was he was saying the first time. I found reading the post very valuable, because of the way Eliezer is constantly looking for his actions to be bound by laws of reasoning, which helps me see the sharp corners and edges of what honesty is, but I have to admit the post is very abstract, and even for many excellent people and close friends, I couldn't hand this essay to them and be confident it wouldn't (on net) confuse them. Getting some of these abstract chains of reasoning right is a tough high-wire act, and while Eliezer does a good job and I learned useful things, I do not expect I personally could have a very safe conversation with someone else while framing our conversation as an attempt to stick to the rules of consistent object-level glomarization when discussing meta-level principles. I feel like I am likely going to be in situations where the term 'meta-honesty' is common knowledge, but the implementation details aren't. I think that I can have the conversations this post is trying to have, but that if we try to speak in the same way that this post speaks, I expect we'll all regularly hit very fraught double-illusions of transparency. In such a situation people will perceive a large number of norm violations that are actually linguistic confusions. Linguistic confusions are not strongly negative in most domains, merely a nuisance, but in prosecuting norm violations, especially when doing so quickly, they have grave consequences. You shouldn't try to build communal norms on such shaky foundations. What happens when we have common knowledge of the terminology but not the substance? Let me be more concrete about this. I think that people can have these conversations, if they put in the effort each time to specify what they're talking about. "Would you lie in this situation?" "Er, feels like you're asking an awfully specific question. Of course I'm happy to answer in general though." “Does <this> feel like a general example?” “Nearly. Is <this nearby example> okay for you?” “Yeah. What would you do there?” “Happy to answer. <This.>” Rather than "Seems like you just lied to that person." "I don't think so." "Well, let’s talk. Are you meta-honest?" "Yeah." "Would you lie in <this> nearby situation?" "Hmm, I don't think I have to tell you." "Huh? The whole point of meta honesty is that you have to tell me." "Well, I can speak generally, but I should glomarise if I think you're getting too object-level." "Can you give me a definition of object-level here?" "Uh, I feel a bit lost." "I think you're being evasive on the meta-level, and you're never supposed to do that." When both people are trying to define terms that neither of them invented and that both are new to... well, I don't think this is a conversation we want to trust norm-disputes to. Another framing: If anyone ever says “I’m not honest, but I’m meta-honest” in a social setting where we didn’t already have common knowledge about how meta-honesty is kinda like a reflectively stable form of high-effort honesty, then I would mostly expect the new common knowledge to be “We here don’t require everyone to stick to the deontological rule of being honest.” I think even if everyone in the room had read Eliezer’s post it wouldn’t change the situation. The main point for me here is that the post is not intuitive, and we can’t currently expect to build common knowledge of the ideas, common knowledge of the meaning of the language, or expect to be able to build subsequent norm-disputes built around it. To give a contrast, I think a post like Arguments About Fast Takeoff is fine to be part of the common knowledge pool from this perspective, because it's not trying to define terms to build a new norm around. To point to the same thing: If Eliezer's post was simply "Here are a bunch of conversations I have with myself about honesty" and they were a bunch of dialogues with no new term-and-norm proposed, I'd feel safer that people didn't feel they were allowed to expect others to understand exactly what they meant. But I think, from the perspective of advocacy for a new term-and-norm, this post doesn't hit the (admittedly very high) bar for making it clear and transparent to all who read it. The problem here is indeed the idea of automatic norms (from Robin Hanson) that Eliezer discusses. Linguistic confusion around takeoff speeds is not a problem (I mean, in the long-run it's an existential catastrophe, but in the short run it's still an interesting conversation), but linguistic confusion over honesty norms can spiral into quite an escalation of perceived norm violations, which can be very damaging. While I think this post managed to say surprising and useful things about the rules of honesty, I think this post has not hit the (very high) bar of making sure the readers have common knowledge of a new language and norm. I think, if this post were to be included in the "Best of 2018" sequence and book, it would be very valuable to have just a few paragraphs accompanying paragraphs saying that when you ask "Can we operate under the norms of meta-honesty?" you must accept "I'd like to taboo the term 'meta-honesty', because I'm not sure we'll be talking about the same thing if we use that term.” I'm not saying "we should renounce meta-honesty". I view it as a valuable idea, in its early stages of being communicated. The idea has been discovered, but the work to make it shared amongst us all has only just begun. As people re-explain it, and flesh it out, I have hope it will become a simple idea for LessWrongers to build shared norms around. Takeaways From This Review 1. Being honest is hard, and there are many weird and difficult edge-cases, such as those including context failures, negotiating with powerful institutions, politicised narratives, and compute limitations. 2. On top of the rule of trying very hard to be honest, Eliezer's post offers an additional general rule for navigating the edge cases. The rule is that when you’re having a general conversation all about the sorts of situations you would and wouldn’t lie, you must be absolutely honest. You can explicitly not answer questions if it seems necessary, but you must never lie. 3. I think this rule is a good extension of the general principle of honesty, and appreciate Eliezer's theoretical arguments for why this rule is necessary. 4. Eliezer’s post introduces some new terminology for discussions of honesty - in particular, the term 'meta-honesty' as the rule instead of 'honesty'. 5. If the term 'meta-honesty' is common knowledge but the implementation details aren't, and if people try to use it, then they will perceive a large number of norm violations that are actually linguistic confusions. Linguistic confusions are not strongly negative in most fields, merely a nuisance, but in discussions of norm-violation (e.g. a court of law) they have grave consequences, and you shouldn't try to build communal norms on such shaky foundations. 6. I and many other people this post was directed at, find it requires multiple readings to understand, so I think that if everyone reads this post, it will not be remotely sufficient for making the implementation details common knowledge, even if the term can become that. 7. In general, I think that everyone should make sure it is acceptable, when asking "Can we operate under the norms of meta-honesty?" for the other person to reply "I'd like to taboo the term 'meta-honesty', because I'm not sure we'll be talking about the same thing if we use that term." 8. This is a valuable bedrock for thinking about the true laws, but not currently at a stage where we can build simple deontological communal rules around. I’d like to see more posts on both fronts. So, if you'd like to dive into all the juicy details, I recommend you read Meta-Honesty: Firming Up Honesty Around Its Edge-Cases To read about how being meta-honest plays out, read sections 6 and 7 (the hardest sections, but the most useful for me). For the theoretical ideas that went into it, see sections 1-4. For counterarguments to the norm, see sections 5 and 8. Footnote [1] Subroutine: Check for literal truth of words. Discuss ### Were vaccines relevant to 20th century US mortality improvements? 10 декабря, 2019 - 03:13 Published on December 10, 2019 12:13 AM UTC I heard a vaccine skeptic claim that “90% of the decline in infectious disease mortality in the 20th century in the US was due to factors other than vaccines.” I wondered, is that right? My guess is yes—but at the same time, I think this is very misleading. That statistic makes it sounds like vaccines just aren’t very important to health—sort of a sideshow in the fight against infectious disease. But here’s what the stat leaves out, and why vaccines still matter: First, the greatest victory of vaccines was over smallpox. Smallpox vaccination was invented in 1796, and other immunization techniques were in use in England and America as early as 1721, so by 1900, immunization had already been fighting smallpox for well over 100 years. Smallpox is also the only disease we have ever completely eradicated—wiped off the face of the earth—and it was only possible because of vaccines. But by the 20th century, most of what remained to be done here was outside the US. So starting the clock in 1900, and restricting to the US, carves out most of the progress against smallpox. Second, if we look just at the US in the 20th century, one of the greatest victories of vaccines was over polio. But the devastation of polio wasn’t just death—it was paralysis. Ten to twenty times more people were paralyzed by polio than died from it (especially after the “iron lung”). Unlike some other diseases, we weren’t able to fight polio with better sanitation or hygiene—in fact, it is believed that improved cleanliness caused the polio epidemics of the late 1800s and early 1900s. (Basically, before good sanitation, most people were exposed to polio in infancy, when they still had leftover immunity from their mothers, and when the disease is less likely to cause paralysis. Cleaner water led to a first exposure later in life, which led to a much worse disease.) So a vaccine was really our only weapon. Third, in general, pharmaceuticals have been a bit ahead of vaccines. For some diseases, such as diphtheria and tuberculosis, an antitoxin or antibiotic was available before a vaccine was. This is also basically the story with influenza/pneumonia. Influenza is a viral disease that often results in an opportunistic infection of bacterial pneumonia. The pneumonia is what kills you. So antibiotics for pneumonia could reduce mortality ultimately caused by influenza. But just because these diseases could be treated with antibiotics (or antitoxins) doesn’t mean vaccines weren’t useful or valuable. Do you really want to wait to get a disease, and then treat it? Isn’t prevention better than cure? Are you totally fine with risking a potentially fatal infection just because drugs exist? What about resistant strains? And what would happen to antibiotic resistance, if we didn’t have vaccines and had to treat a much larger number of patients? Fourth, looking only at mortality also simply ignores a variety of less common and/or less deadly diseases that are still important, such as chickenpox, hepatitis, mumps, rubella, and tetanus. True, these don’t add up to pneumonia or TB. But should we then just write them off? Coming back to the original claim: Good data on these questions is non-trivial to come by and to analyze. But in the US in 1900, the top killers among infectious diseases were pneumonia, tuberculosis, various forms of gastrointestinal infections, and (distant fourth) diphtheria. The “90% not due to vaccines” claim is plausible to me because, for a variety of reasons, vaccines may not have been the first thing to drastically reduce mortality from these specific causes: • Pneumonia and gastroenteritis can be caused by a wide variety of germs (vaccines only protect against specific germs) • Tuberculosis, and bacterial forms of pneumonia and gastroenteritis, can be fought with antibiotics • Diphtheria had antitoxins since the 1890s; the vaccine wasn’t available until the 1920s • The tuberculosis vaccine has proved difficult to develop; even the best one we have today has varying efficacy • Similarly, influenza mutates so fast that it’s impossible to develop and administer a vaccine for every strain of it; the annual “flu shot” only protects against the strain that we guess will be most prevalent that year You could look at all those facts and say that vaccines are overrated. And perhaps antibiotics deserve the highest honors in the fight against infectious disease. But it would be a mistake to discount or dismiss vaccines, for the following reasons: 1. They are our only highly effective weapon against highly contagious viral diseases, such as polio, influenza, measles, and (in the past) smallpox. 2. They complement pharmaceuticals, providing defense in depth. Vaccines are prevention; drugs are cure. You want both. 3. Drug resistance is real and presents a risk; and the more we have to use drugs the worse it will get. 4. Suffering from a disease but not dying from it is still suffering. The vaccines the CDC recommends for routine immunization do not include diseases that have been successfully reduced by sanitation or pest control, such as yellow fever, typhoid fever, and cholera; or by eradication (smallpox); or those that are otherwise rare (anthrax). They basically only recommend vaccines that are highly effective or are for highly contagious diseases; in most cases both. The flu shot, which is only partially effective, and tetanus, which is not highly contagious, are both common enough in the US to be warranted. Bottom line: the 90% claim is probably true, but: • Vaccines are still very important and deserve major credit in the fight against infectious disease • You should still get your shots (and vaccinate your kids). I’m working on a better quantitative analysis to answer: which diseases were the worst, and which methods to fight them deserve most credit? I’ll post here when I have more info. In the meantime, if you have any pointers to good papers or data sources on this, let me know. Discuss ### Bayesian examination 9 декабря, 2019 - 23:43 Published on December 9, 2019 7:50 PM UTC A few months ago, Olivier Bailleux, a Professor of computer science and reader of my book on Bayesianism, sent me an email. He suggested to apply some of the ideas of the book to examine students. He proposed Bayesian examination. I believe it to be a brilliant idea, which could have an important impact on how many people think. At least, I think that this is surely worth sharing here. tl;dr Bayesian examinations seem very important to deploy because they incentivize both probabilistic thinking and intellectual honesty. Yet, as argued by Julia Galef in this talk, incentives seem critical to change our thinking habits. Let's take an example Where is the International Olympic Committee? 1. Geneva 2. Lausanne 3. Zurich 4. Lugano Quite often, students are asked to select one of the four possible answers. But this is arguably pretty bad, for several reasons: - It makes impossible to distinguish a student who has a hunch from a student who really studied and knew the answer. - It gives students the habit of self-identifying with a single answer. - It normalizes deterministic question answering. - It motivates students to defend the answer they gave (which encourages the motivated reasoning fallacy...). Instead, Bayesian examination demands that students provide probabilistic answers. In other words they will have to provide percentage for each answer. In our case, a student, call her Alice, might thus answer 1. 33% 2. 33% 3. 33% 4. 1% Alice would essentially be formalizing the sentence "I really don't know but I would be very surprised if Lugano was the right answer". Another student, let's call him Bob, might answer 1. 5% 2. 40% 3. 50% 4. 5% Bob might be having in mind something like "I know that FIFA and CIO are in Zurich and Lausanne, but I don't remember which is where; though Zurich is larger so it would make sense for CIO to be in Zurich rather than Lausanne". Spoiler: the answer turns out to be Lausanne. Why naive notation is bad Now, how would such an exam be scored? One intuitive idea could be that Alice should thus get 0.33 points, while Bob should get 0.4 points. Denoting .mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} qi the probability assigned by a student to answer i, and i∗ the right answer, this would correspond to giving the student a score equals to S=qi∗. This would not be a great idea though. The reason for this has to do with incentives. Indeed, it turns out that if the above figures are the credences pi of Alice and Bob, then Alice and Bob would be incentivized to maximize their expected scores E[S]=p1q1+p2q2+p3q3+p4q4. It turns out that this maximization leads to the following answers. For Alice: 1. Credence p1=33% in Geneva, but answers q1=100%. 2. Credence p2=33% in Lausanne, but answers q2=0%. 3. Credence p3=33% in Zurich, but answers q3=0%. 4. Credence p4=33% in Lugano, but answers q4=0%. For Bob: 1. Credence p1=5% in Geneva, but answers q1=0%. 2. Credence p2=40% in Lausanne, but answers q2=0%. 3. Credence p3=50% in Zurich, but answers q3=100%. 4. Credence p4=5% in Lugano, but answers q4=0%. In other words, this naive scoring incentivizes the exaggeration of beliefs towards deterministic answer. This is very, very, very, very, very bad (sorry I'm a bit of Bayesian extremist!). This favors polarization, rationalization, groupism and so many other root causes of poor debating. Indeed, while students may not find out consciously that this exaggeration strategy is optimal, we should expect them to eventually try it and not unconsciously notice that this is not so bad. In particular, this prevents them from valuing the extra-effort of probabilistic thinking. Fortunately, there are better scoring rules. Incentive-compatible scoring rules An incentive-compatible scoring rule is called a proper scoring rule. But I'm not keen on the terminology, as it's not transparent, so I'll stick with incentive-compatible scoring rule. Such incentive-compatible scoring rules are such that truth-telling (or rather "credence-telling") is incentivized. There are several incentive-compatible scoring rules, like the logarithmic scoring rule (S=ln(qi∗)) or the spherical scoring rule (S=qi∗/||q||2). But I think that the most appropriate one may be the quadratic scoring rule, because it is the simplest and easiest for students to verify. In our case, given that the right answer was Lausanne, the score of a student who answered q1,q2,q3 and q4 is S=1−(q21+(1−q2)2+q23+q24). In other words, for each possibility i, the student loses the square of the distance between his answer qi and the true answer (0% or 100%). In our case, Alice would win S=0.3332 points, while Bob would win S=0.585 points. Of course, the right answer q=(0,1,0,0) would win 1 point, while any maximally wrong answer like q=(1,0,0,0) would lose 1 point. Perhaps more interestingly, a maximally ignorant student who answers 25% to each possibility would win S=0.25 points. This is much better than the expected answer of random deterministic guess, which equals E[S]=−0.5. Exaggerated guesses get greatly penalized. In fact, they yield negative points! Formally, the quadratic scoring rule equals S=1−||q−ei∗||22=2qi∗−||q||22, where ei∗ is the basis vector whose entries are all zeros except for the i∗-th coordinate, which is 1. If there are n answers, then the maximally ignorant student wins S=1/n, while the random deterministic guesser wins an expectation of E[S]=−1+(2/n). Why this is important Because wrong answers are much more penalized than acknowledging ignorance, students who aim to maximize their scores will likely eventually learn, consciously or not, that guessing deterministic answers is just wrong. They may even learn the habit of second-guessing their intuitions, and to add uncertainty to their first guesses. In terms of rationality, this seems like a huge deal! Perhaps equally importantly, such Bayesian examinations incentivize students to take on probabilistic reasoning. Students may thereby learn to constantly measure appropriately their levels of confidence, and to reason with (epistemological) uncertainty. As an aspiring Bayesian, this is the part I'm most excited about! Finally, and probably even more importantly, such examinations incentivize intellectual honesty. This is the habit of trying to be honest, not only with others, but also with ourselves. It's sometimes said that "a bet is a tax on bullshit", as argued by Alex Tabarrok. Arguably, Bayesian examinations are even better than a bet. Indeed, in (important) exams, we might be making an even bigger effort than when we put our money where our mouth! In case you're still not convinced by the importance of intellectual honesty, I highly recommend this talk by Julia Galef or her upcoming book (as well as, say, Tetlock and Gardner's Superforecasting book). Where to go from here I haven't had the chance to test these ideas though. I wonder how students and teachers will feel about it. I suspect some pushback early on. But I would also bet that students may eventually appreciate it. To find out, I guess this really needs to be tested out there! One particular platform that could be a great first step is in MOOCs and other online websites where people enter their answers electronically. If you happen to be working in such areas, or to know people working in these areas, I think it would be great to encourage a trial of Bayesian examinations! And if you do, please send me feedbacks. And please let me test your exams as well :P Still another approach would be to develop an app to record Bayesian bets that we make, and to compute our incentive-compatible (quadratic?) scores. Gamifying the app might make it more popular. If anyone is keen on developing such an app, I'd be more than eager to test it, and to train my own Bayesian forecasting abilities! PS : If you're French-speaking (or motivated to read subtitles), you can also check out the video I made on the same topic. Discuss ### Is Rationalist Self-Improvement Real? 9 декабря, 2019 - 20:11 Published on December 9, 2019 5:11 PM UTC Cross-posted from Putanumonit. Basketballism Imagine that tomorrow everyone on the planet forgets the concept of training basketball skills. The next day everyone is as good at basketball as they were the previous day, but this talent is assumed to be fixed. No one expects their performance to change over time. No one teaches basketball, although many people continue to play the game for fun. Snapshots of a Steph Curry jump shot. Image credit: ESPN. Geneticists explain that some people are born with better hand-eye coordination and are thus able to shoot a basketball accurately. Economists explain that highly-paid NBA players have a stronger incentive to hit shots, which explains their improved performance. Psychologists note that people who take more jump shots each day hit a higher percentage and theorize a principal factor of basketball affinity that influences both desire and skill at basketball. Critical theorists claim that white men’s under-representation in the NBA is due to systemic oppression. Papers are published, tenure is awarded. New scientific disciplines emerge and begin studying jump shots more systematically. Evolutionary physiologists point out that our ancestors threw stones more often than they tossed basketballs, which explains our lack of adaptation to that particular motion. Behavioral kinesiologists describe systematic biases in human basketball, such as the tendency to shoot balls with a flatter trajectory and a lower release point than is optimal. When asked by aspiring basketball players if jump shots can be improved, they all shake their heads sadly and say that it is human nature to miss shots. A Nobel laureate behavioral kinesiologist tells audiences that even after he wrote a book on biases in basketball his shot did not improve much. Someone publishes a study showing that basketball performance improves after a one-hour training session with schoolchildren, but Shott Ballexander writes a critical takedown pointing out that the effect wore off after a month and could simply be random noise. The field switches to studying “nudges”: ways to design systems so that players hit more shots at the same level of skill. They recommend that the NBA adopt larger hoops. Papers are published, tenure is awarded. Then, one day, a woman sits down to read those papers who is not an academic, just someone looking to get good at basketball. She realizes that the lessons of behavioral kinesiology can be used to improve her jump shot, and practices releasing the ball at the top of her jump from above the forehead. Her economist friend reminds her to give the ball more arc. As the balls start swooshing in, more people gather at the gym to practice shooting. They call themselves Basketballists. Most people who walk past the gym sneer at the Basketballists. “You call yourselves Basketballists and yet none of you shoots 100%”, they taunt. “You should go to grad school if you want to learn about jump shots.” Some of Basketballists themselves begin to doubt the project, especially since switching to the new shooting techniques lowers their performance at first. “Did you hear what the Center for Applied Basketball is charging for a training camp?” they mutter. “I bet it doesn’t even work.” Within a few years, some dedicated Basketballists start talking about how much their shot percentage improved and the pick-up tournaments they won. Most people say it’s just selection bias, or dismiss them by asking why they can’t outplay Kawhi Leonard for all their training. The Basketballists insist that the training does help, that they really get better by the day. But how could they know? AsWrongAsEver A core axiom of Rationality is that it is a skill that can be improved with time and practice. The very names Overcoming Bias and Less Wrong reflect this: rationality is a vector, not a fixed point. A core foundation of Rationality is the research on heuristic and biases led by Daniel Kahneman. The very first book in The Sequences is in large part a summary of Kahneman’s work. Awkwardly for Rationalists, Daniel Kahneman is hugely skeptical of any possible improvement in rationality, especially for whole groups of people. In an astonishing interview with Sam Harris, Kahneman describes bias after bias in human thinking, emotions, and decision making. For every one, Sam asks: how do we get better at this? And for every one, Daniel replies: we don’t, we’ve been telling people about this for decades and nothing has changed, that’s just how people are. Daniel Kahneman is familiar with CFAR but as far as I know, he has not put as much effort himself into developing a community and curriculum dedicated to improving human rationality. He has discovered and described human irrationality, mostly to an audience of psychology undergrads. And psychology undergrads do worse than pigeons at learning a simple probabilistic game so we shouldn’t expect them to learn rationality just by reading about biases. Perhaps if they started reading Slate Star Codex… Alas, Scott Alexander himself is quite skeptical of Rationalist self-improvement. He certainly believes that Rationalist thinking can help you make good predictions and occasionally distinguish truth from bullshit, but he’s unconvinced that the underlying ability can be improved upon. Scott is even more skeptical of Rationality’s use for life-optimization. I told Scott that I credit Rationality with a lot of the massive improvements in my financial, social, romantic, and mental life that happened to coincide with my discovery of LessWrong. Scott argued that I would do equally well in the absence of Rationality by finding other self-improvement philosophies to pour my intelligence and motivation into and that these two are the root cause of my life getting better. Scott also seems to have been doing very well since he discovered LessWrong, but he credits Rationality with not much more than being a flag that united the community he’s part of. So: on one side are Yudkowsky, CFAR, and several Rationalists, sharing the belief that Rationality is a learnable skill that can improve the lives of most seekers who step on the path. On the other side are Kahneman, Scott, several other Rationalists, and all anti-Rationalists, who disagree. When I surveyed my Twitter followers, the results distributed somewhat predictably: The optimistic take is that RSI works for most people if they only tried it. The neutral take is that people are good at trying self-improvement philosophies that would work for them. The pessimistic take is that Rationalists are deluded by sunk cost and confirmation bias. Who’s right? Is Rationality trainable like jump shots or fixed like height? Before reaching any conclusions, let’s try to figure out how why so many smart people who are equally familiar with Rationality disagree so strongly about this important question. Great Expectations An important crux of disagreement between me and Scott is in the question of what counts as successful Rationalist self-improvement. We can both look at the same facts and come to very different conclusions regarding the utility of Rationality. Here’s how Scott parses the fact that 15% of SSC readers who were referred by LessWrong have made over1,000 by investing in cryptocurrency and 3% made over $100,000: The first mention of Bitcoin on Less Wrong, a post called Making Money With Bitcoin, was in early 2011 – when it was worth 91 cents. Gwern predicted that it could someday be worth “upwards of$10,000 a bitcoin”. […]This was the easiest test case of our “make good choices” ability that we could possibly have gotten, the one where a multiply-your-money-by-a-thousand-times opportunity basically fell out of the sky and hit our community on its collective head. So how did we do?I would say we did mediocre. […]Overall, if this was a test for us, I give the community a C and me personally an F. God arranged for the perfect opportunity to fall into our lap. We vaguely converged onto the right answer in an epistemic sense. And 3 – 15% of us, not including me, actually took advantage of it and got somewhat rich.

Here’s how I would describe it:

Of the 1289 people who were referred to SSC from LessWrong, two thirds are younger than 30, a third are students/interns or otherwise yet to start their careers, and many are for other reasons too broke for it to be actually rational to risk even $100 on something that you saw recommended on a blog. Of the remainder, the majority were not around in the early days when cryptocurrencies were discussed — the median “time in community” on LessWrong surveys is around two years. In any case, “invest in crypto” was never a major theme or universally endorsed in the Rationalist community.Of those that were around and had the money to invest early enough, a lot lost it all when Mt. Gox was hacked or when Bitcoin crashed in late 2013 and didn’t recover until 2017 or through several other contingencies.If I had to guess the percent of Rationalists who were even in a position to learn about crypto on LessWrong and make more than$1,000 by following Rationalist advice, I’d say it’s certainly less than 50%. Maybe not much larger than 15%.Only 8% of Americans own cryptocurrency today. At the absolute highest end estimate, 1% of Americans, and 0.1% of people worldwide, made >\$1,000 from crypto. So Rationalists did at least an order of magnitude better than the general population, almost as well as they could’ve done in a perfect world, and also funded MIRI and CFAR with Bitcoin for years ahead. I give the community an A and myself an A.

In an essay called Extreme Rationality: It’s Not That Great Scott writes:

Eliezer writes:
The novice goes astray and says, “The Art failed me.”
The master goes astray and says, “I failed my Art.”Yet one way to fail your Art is to expect more of it than it can deliver.

Scott means to say that Eliezer expects too much of the art in demanding that great Rationalist teachers be great at other things as well. But I think that expecting 50% of LessWrongers filling out a survey to have made thousands of dollars from crypto is setting the bar far higher than Eliezer’s criterion of “Being a math professor at a small university who has published a few original proofs, or a successful day trader who retired after five years to become an organic farmer, or a serial entrepreneur who lived through three failed startups before going back to a more ordinary job as a senior programmer.”

Akrasia

Scott blames the failure of Rationality to help primarily on akrasia.

One factor we have to once again come back to is akrasia. I find akrasia in myself and others to be the most important limiting factor to our success. Think of that phrase “limiting factor” formally, the way you’d think of the limiting reagent in chemistry. When there’s a limiting reagent, it doesn’t matter how much more of the other reagents you add, the reaction’s not going to make any more product. Rational decisions are practically useless without the willpower to carry them out. If our limiting reagent is willpower and not rationality, throwing truckloads of rationality into our brains isn’t going to increase success very much.

I take this paragraph to imply a model that looks like this:

[Alex reads LessWrong] -> [Alex tries to become less wrong] -> [akrasia] -> [Alex doesn’t improve].

I would make a small change to this model:

[Alex reads LessWrong] -> [akrasia] -> [Alex doesn’t try to become less wrong] -> [Alex doesn’t improve].

A lot of LessWrong is very fun to read, as is all of SlateStarCodex. A large number of people on these sites, as on Putanumonit, are just looking to procrastinate during the workday, not to change how their mind works. Only 7% of the people who were engaged enough to fill out the last LessWrong survey have attended a CFAR workshop. Only 20% ever wrote a post, which is some measure of active rather than passive engagement with the material.

In contrast, one person wrote a sequence on trying out applied rationality for 30 days straight: Xiaoyu “The Hammer” He. And he was quite satisfied with the result.

I’m not sure that Scott and I disagree much, but I didn’t get the sense that his essay was saying “just reading about this stuff doesn’t help, you have to actually try”. It also doesn’t explain was he was so skeptical about me crediting my own improvement to Rationality.

Akrasia is discussed a lot on LessWrong, and applied rationality has several tools that help with it. What works for me and my smart friends is not to try and generate willpower but to use lucid moments to design plans that take a lack of willpower into account. Other approaches work for other people. But of course, if someone lacks the willpower to even try and take Rationality improvement seriously, a mere blog post will not help them.

3% LessWrong

Scott also highlights the key sentence in his essay:

I think it may help me succeed in life a little, but I think the correlation between x-rationality and success is probably closer to 0.1 than to 1.

What he doesn’t ask himself is: how big is a correlation of 0.1?

Here’s the chart of respondents to the SlateStarCodex survey, by self-reported yearly income and whether they were referred from LessWrong (Scott’s criterion for Rationalists).

And here’s the same chart with a small change. Can you notice it?

For the second chart, I increased the income of all rationalists by 25%.

The following things are both true:

• When you eyeball the group as a whole, the charts look identical. A 25% improvement for a quarter of the people in a group you observe is barely noticeable. The rich stayed rich, the poor stayed poor.
• If your own income increased 25% you would certainly notice it. And if the increase came as a result of reading a few blog posts and coming to a few meetups, you would tell everyone you know about this astounding life hack.

The correlation between Rationality and income in Scott’s survey is -0.01. That number goes up to a mere 0.02 after the increase. A correlation of 0.1 is absolutely huge, it would require tripling the income of all Rationalists.

The point isn’t to nitpick Scott’s choice of “correlation = 0.1” as a metaphor. But every measure of success we care about, like impact on the world or popularity or enlightenment, is probably distributed like income is on the survey. And so if Rationality made you 25% more successful it wouldn’t be as obviously visible as Scott thinks it would be — especially since everyone pursues a different vision of success. In this 25% world, the most and least successful people would still be such for reasons other than Rationality. And in this world, Rationality would be one of the most effective self-improvement approaches ever devised. 25% is a lot!

Of course, the 25% increase wouldn’t happen immediately. Most people who take Rationality seriously have been in the community for several years. You get to 25% improvement by getting 3% better each year for 8 years.

You know what feels crappy? 3% improvement. You busted your ass for a year, trying to get better at dating, at being less of an introvert, at self-soothing your anxiety – and you only managed to get 3% better at it.If you worked a job where you put in that much time at the office and they gave you a measly 3% raise, you would spit in your boss’s face and walk the fuck out.
And, in fact, that’s what most people do: quit. […]The model for most self-improvement is usually this:
* You don’t have much of a problem
* You found The Breakthrough that erased all the issues you had
* When you’re done, you’ll be the opposite of what you were. Used to be bad at dating? Now you’ll have your own personal harem. Used to be useless at small talk? Now you’re a fluent raconteur.Which, when you’ve agonized to scrape together a measly 3% improvement, feels like crap. If you’re burdened with such social anxiety that it takes literally everything you have to go out in public for twenty minutes, make one awkward small talk, and then retreat home to collapse in embarrassment, you think, “Well, this isn’t worth it.”But most self-improvement isn’t immediate improvement, my friend.
It’s compound interest.

I think that Rationalist self-improvement is like this. You don’t get better at life and rationality after taking one class with Prof. Kahnemann. After 8 years of hard work, you don’t stand out from the crowd even as the results become personally noticeable. And if you discover Rationality in college and stick with it, by the time you’re 55 you will be three times better than what you would have been if you hadn’t compounded these 3% gains year after year, and everyone will notice that.

What’s more, the outcomes don’t scale smoothly with your level of skill. When rare, high leverage opportunities come around, being slightly more rational can make a huge difference. Bitcoin was one such opportunity; meeting my wife was another such one for me.  I don’t know what the next one will be: an emerging technology startup? a political upheaval? cryonics? I know that the world is getting weirder faster, and the payouts to Rationality are going to increase commensurately.

Probability theory in general, and Bayesianism in particular, provide a coherent philosophical foundation for not being an idiot.Now in general, people don’t need coherent philosophical foundations for anything they do. They don’t need grammar to speak a language, they don’t need classical physics to hit a baseball, and they don’t need probability theory to make good decisions. This is why I find all the “But probability theory isn’t that useful in everyday life!” complaining so vacuous.“Everyday life” means “inside your comfort zone”. You don’t need theory inside your comfort zone, because you already navigate it effortlessly. But sometimes you find that the inside of your comfort zone isn’t so comfortable after all (my go-to grammatical example is answering the phone “Scott? Yes, this is him.”) Other times you want to leave your comfort zone, by for example speaking a foreign language or creating a conlang.When David says that “You can’t possibly be an atheist because…” doesn’t count because it’s an edge case, I respond that it’s exactly the sort of thing that should count because it’s people trying to actually think about an issue outside their comfort zone which they can’t handle on intuition alone. It turns out when most people try this they fail miserably. If you are the sort of person who likes to deal with complicated philosophical problems outside the comfortable area where you can rely on instinct – and politics, religion, philosophy, and charity all fall in that area – then it’s really nice to have an epistemology that doesn’t suck.

If you’re the sort of person for whom success in life means stepping outside the comfort zone that your parents and high school counselor charted out for you, if you’re willing to explore spaces of consciousness and relationships that other people warn you about, if you compare yourself only to who you were yesterday and not to who someone else is today… If you’re weird like me, and if you’re reading this you probably are, I think that Rationality can improve your life a lot.

But to get better at basketball, you have to actually show up to the gym.