# Новости LessWrong.com

A community blog devoted to refining the art of rationality
Обновлено: 53 минуты 46 секунд назад

### Scope Insensitivity Judo

19 июля, 2019 - 20:33
Published on July 19, 2019 5:33 PM UTC

It's easy to bemoan scope insensitivity, a special case of that phenomenon where we mere humans end up caring more about the death of one person than one hundred, better remember the last bite of a meal than the first dozen, and think less is more and more is less. After all, if we didn't neglect scope we would be more rational, and so maybe happier and healthier, living in a world were everyone got more of what they wanted, since without scope insensitivity it wouldn't be so hard to convince people to help those far away who need more than those nearby who need less. But scope insensitivity is what we've got, so we have to learn to live with it.

Luckily there's plenty of reason to think we can take advantage of scope insensitive because people have already discovered ways to make the best of other forms of extension neglect. For example, adapting to duration neglect is something most people learn early on by adopting heuristics like "save the best for last" and "do the hardest part first". Salespersons and motivational speakers alike learn to exploit base rate neglect, sample size neglect, and the conjunction fallacy to convince people to do what they otherwise might not. And designers of all kinds of systems can mold incentives to work with rather than against human nature. Thus it stands to reason we can use our natural scope insensitivity to do more than fail to multiply.

I'll consider one such use case here, namely a practice of using scope insensitivity to prepare ourselves for high-stakes situations in low-stakes ones. This is a kind of scope insensitivity judo, or "gentle way", and just like in the martial art, we'll redirect the strength of our "opponent" to transform it into an unintended ally.

Dwell in the Dojo

Life is full of high-stakes situations: job interviews, first dates, nuclear missile crises. These generally feel like one-shot scenarios—there's one chance to get it right and if we fail all is lost. To wit, if we don't say the right things we'll lose our shot at that job forever, if we don't put out the right vibes that person will never fall in love with us, and if we push the big red button there'll be no second chances for anything.

We can make these multi-shot scenarios pretty easily with training, and there's some value in practicing interviewing skills, going on many dates so no one date matters very much, and running war games. These are all training methods that take something high-stakes and make it low-stakes so you feel free to experiment. That's one way to learn: by creating a safe laboratory where we can explore more before we prune.

That's not what I'm suggesting we do, though. In the dojo of scope insensitivity judo, we practice the way of getting into low-stakes scenarios that feel high-stakes so we are prepared generally to handle really high-stakes scenarios when we encounter them. We do this by taking advantage of the way our minds mistakenly believe many low-stakes scenarios are high-stakes ones because they push against beliefs and behaviors that were evolutionarily or historically adaptive but no longer are.

Consider these examples from my own life, drawn from my zen practice:

• I asked if I could bring a cushion from home for a retreat. I was told yes. I brought it. The cushion was orange, the zendo's cushions were black, it stuck out, and I was told I couldn't use my cushion.
• I complied, but I was immediately caught by thoughts like "but you told me I could use my cushion" and "now my meditation will be worse because I'll be less comfortable" and "I'm not as good a zen student as I thought".
• I felt embarrassed, defensive, let down, and defeated. I felt like a failure, like I was 2nd grade Gordy again getting in trouble for being weird.
• Of course, stepping back, we can see this was a very low-stakes situation: I just switched cushions and got on with the retreat! But it felt high-stakes at the time because it pushed me in ways that might have been adaptive in some high-stakes situations, either in my personal past or within my cultural or evolutionary environment. For our ancestors, this kind of mistake could have meant loss of prestige and thus loss of resources and thus marginal loss of reproductive and survival opportunities. Lucky for me it was just about a cushion in the zendo!
• I was sitting half lotus during a long meditation period, and after about 40 minutes my legs hurt in a way that I worried was injuring them by continuing, so I uncrossed my legs and sat with them pulled up towards my chest to give them a rest. In the middle of this my teacher walked into the zendo and saw me, and came over to correct me, saying I couldn't sit like that and had to either sit crosslegged or in a chair.
• I sort of complied: I instead took the option to do brief walking meditation before returning to sitting. I was caught by thoughts like "you didn't see how I had been sitting" and "you didn't know the kind of danger I was in" and "I must have stayed sitting out-of-form because the pain was so bad it temporarily addled my mind".
• I felt embarrassed, ashamed, and defensive and also a little indignant.
• This was also a pretty low-stakes situation: I walked for 10 minutes, came back and sat for the rest of the period, and it was never mentioned again. I didn't lose any of my positions or responsibilities, and my practice continued on as strong as ever. But it felt high-stakes because I had been caught out and corrected in front of others, and maybe they would think less of me. As best I can tell, they did not.
• A new person came to our Saturday morning practice period for the first time. I was work leader that week, and when it came time to hand out assignments I assigned her to clean the zendo under my supervision. I was later corrected by the person who trained me as work leader that I shouldn't have given her that assignment because new people should get simple tasks like sweeping.
• I was immediately somewhat defensive. Cleaning the zendo was the job I had been assigned when I first came to the zen center, so I thought it was the right thing to do. I said as much.
• In addition to being defensive, I felt like I had been let down by my trainer not telling me this before, and I also felt I had the excuse that it worked out fine.
• Once again, this was pretty low-stakes: she cleaned the zendo well, I got new information, and I changed how I hand out work assignments. But my behavior indicates I thought it was high-stakes enough to be worth some back-and-forth and argument or defense of my position and to put myself in opposition to another person to save face. I had wanted to do a good job at being work leader, and felt threatened by the correction, leading me to escalate my response.

I drew these from my zen practice because the zen center is like a laboratory where we specialize in studying the self, and so I had more chance to examine these events and remember them than the many similar daily occurrences that happen throughout the rest of my life. Also they are less personal and raw than the times I blew low-stakes situations out-of-proportion and didn't learn from them at work, with family, and among friends. But hopefully those are enough for you to start to see the pattern: scope insensitivity means we often treat low-stakes situations like high-stakes situations, and we can take advantage of that to use them as training scenarios for genuine high-stakes events if we allow ourselves the space to stop and take a step back to consider what we're doing.

The Way of Scope Insensitivity

You can begin to practice with scope insensitivity yourself right away, because the world is constantly presenting you with low-stakes scenarios that feel high-stakes. The more anxious, depressed, or frustrated you generally are, the more you are likely you are treating low-stakes situations as high-stakes and so you will have even more opportunities to practice scope insensitivity judo than people who are more calm and equanimous.

The first part of the practice is to notice and stop. Notice when you feel like you are in a high-stakes situation. Then stop for a few breaths to examine it. Don't worry if you fail at first; learning to notice is hard if you're not already skilled at it, and even when you are skilled it's still easy to get so caught up that we forget to really look.

When you catch one of these situations, consider whether it is really high-stakes, or if you just believe it is due to scope insensitivity. If it's really low-stakes, this is a great opportunity to experiment and practice with dealing with these situations and the factors that cause them to feel high-stakes. If you're sure it's really high-stakes, that's even better, though you'll want to be a bit more cautious in how you proceed.

There are many ways you can explore these situations once you've noticed them arising, and the path you take largely depends on what you are ready for and what resonates with you. I've gotten a lot of milage out of the Immunity to Change framework and working with core beliefs (albeit within the Ordinary Mind zen context rather than a CBT context). You might prefer something that looks more like psychotherapy, various CFAR techniques, Folding, Focusing, Core Transformation, or some kind of debugging. Generally you are looking for a way to integrate what you can see when you step back and look at what's happening in these situations with your immediate reactions, and anything that helps you do that will likely work here.

And then you just keep doing it. You're unlikely to fix your scope insensitivity—that appears to just be part of how human brains work. But you can, through regular practice, retrain yourself to more deftly handle situations that previously felt overwhelming. By developing the skill of flipping what feel like high-stakes situations into low-stakes ones, you'll gain perspective on those situations that allows you to take a more thoughtful, deliberate approach that transcends the worst of our knee-jerk reactions that lead to self-created suffering.

Cross-posted to Map and Territory.

Discuss

### From Laplace to BIC

19 июля, 2019 - 19:52
Published on July 19, 2019 4:52 PM UTC

The previous post outlined Laplace approximation, one of the most common tools used to approximate hairy probability integrals. In this post, we'll use Laplace approximation to derive the Bayesian Information Criterion (BIC), a popular complexity penalty method for comparing models with more free parameters to models with fewer free parameters.

The BIC is pretty simple:

• Subtract k2lnN to penalize for complexity, where N is the number of data points and k is the number of free parameters (i.e. dimension of θ).

Thus: lnP[data|θmax]−k2lnN. Using this magic number, we compare any two models we like.

Let's derive that.

BIC Derivation

As usual, we'll start from P[data|model]. (Caution: don't forget that what we really care about is P[model|data]; we can jump to P[data|model] only as long as our priors are close enough to be swamped by the evidence.) This time, we'll assume that we have N independent data points xi, all with the same unobserved parameters - e.g. N die rolls with the same unobserved biases. In that case, we have

P[data|model]=∫θP[data|θ]dP[θ]=∫θ∏Ni=1P[xi|θ]p[θ]dθ

Next, apply Laplace approximation and take the log.

lnP[data|model]≈∑ilnP[xi|θmax]+lnp[θmax]+k2ln(2π)−12lndet(H)

where the Hessian matrix H is given by

H=d2dθ2lnP[data|θ]|θmax=∑id2dθ2lnP[xi|θ]|θmax

Now for the main trick: how does each term scale as the number of data points N increases?

• The max log likelihood ∑iP[xi|θmax] is a sum over data points, so it should scale roughly proportionally to N.
• The prior density and the k2ln(2π) are constant with respect to N.
• H is another sum over data points, so it should also scale roughly proportionally to N.

Let's go ahead and write H as N∗(1NH), to pull out the N-dependence. Then, if we can remember how determinants scale:

lndet(N∗(1NH))=lndet(1NH)+k∗lnN

so we can re-write our Laplace approximation as lnP[data|model]≈∑ilnP[xi|θmax]+p[θmax]+k2ln(2π)−12lndet(1NH)−k2lnN=lnP[data|θmax]−k2lnN+O(1)

where O(1) contains all the terms which are roughly constant with respect to N. The first two terms are the BIC.

In other words, the BIC is just the Laplace approximation, but ignoring all the terms which don't scale up as the number of data points increases.

When Does BIC Work?

What conditions need to hold for BIC to work? Let's go back through the derivation and list out the key assumptions behind our approximations.

• First, in order to jump from P[model|data] to P[data|model], our models should have roughly similar prior probabilities P[model] - i.e. within a few orders of magnitude.
• Second, in order for any point approximation to work, the posterior parameter distribution needs to be pointy and unimodal - most of the posterior probability mass must be near θmax. In other words, we need enough data to get a precise estimate of the unobserved parameters.
• Third, we must have N large enough that k2lnN (the smallest term we're keeping) is much larger than the O(1) terms we're throwing away.

That last condition is the big one. BIC is a large-N approximation, so N needs to be large for it to work. How large? That depends how big lndet(1NH) is - N needs to be exponentially larger than that. We'll see an example in the next post.

Next post will talk more about relative advantages of BIC, Laplace, and exact calculation for comparing models. We'll see a concrete example of when BIC works/fails.

Discuss

### Against Excessive Apologising

19 июля, 2019 - 18:00
Published on July 19, 2019 3:00 PM UTC

Many people would say that if you realise that you are in the wrong, then you should always apologise. Perhaps, they'd exclude sociopathic situations where this would be used to manipulate you, but that'd be it.

However, it's easy to forget that apologising creates a cost for the person who is apologised to. They have to read your message and perhaps write a reply. This later component is tricky if they aren't convinced that you've made up for it. It reminds them of an experience they might want to forget. Further, it requires them to deal with a topic they may be completely sick and tired of.

If you apologise, it should be because it helps prevent or mend a rift with the other person. You should be extremely cautious about apologising because that's what you think a nice person would do, as those are precisely the situations where you are likely to end up apologising with no benefit to anyone.

Discuss

### [Link] Intro to causal inference by Michael Nielson (2012)

19 июля, 2019 - 15:19
Published on July 19, 2019 12:19 PM UTC

This is a link post for Michael Nielson's "If correlation doesn’t imply causation, then what does?" (2012).

I want to highlight the post for a few reasons:

(1) it is a well-written introduction by an experienced science communicator — Michael is an author of the most famous book on quantum computing;

(2) causal inference is an essential tool for understanding the world;

(3) two recent AI safety papers use causal influence diagrams to (a) understand agent incentives [arXiv, Medium] and (b) to provide a new perspective on some problems in AGI safety [arXiv].

Discuss

### When does adding more people reliably make a system better?

19 июля, 2019 - 07:21
Published on July 19, 2019 4:21 AM UTC

Prediction markets have a remarkable property. They reward correct contrarianism. They incentivise people to disagree with the majority consensus, and be right. If you add more traders to a market, in expectation they price will be more accurate.

More traders means both more fish and more sharks.

(The movie "The Big Short" might be a very sad portrait of the global financial system. But it's still the case that a system in a bad equilibrium with deeply immoral consequences rewarded the outcasts who pointed out those consequences with billions of dollars. Even though socially, no one bothered listening to them, including the US Government who ignored requests by one of the fund managers to share his expertise about the events after the crash.)

Lots of things we care about don't have this property.

• Many social communities decline as more members join, and have to spend huge amounts of effort building institutions and rituals to prevent this.
• Many companies have their culture decline as they hire more, and have to spend an incredible amount of resources simply to prevent this (which is far from getting better as more people join). (E.g. big tech companies can probably have >=5 candidates spend >=10 hours in interviews for a a single position. And that's not counting the probably >=50 candidates for that position spending >=1h.)
• Online forums usually decline with growing user numbers (this happened to Reddit, HackerNews, as well as LessWrong 1.0).

In prediction markets the vetting process is really cheap. You might have to do some KYC, but mostly new people is great. This seems like a really imporant property for a system to have, and something we could learn from to build other such systems.

What other systems have this property?

Discuss

### Becoming a Robust Person, or Organization

19 июля, 2019 - 04:18
Published on July 19, 2019 1:18 AM UTC

Epistemic Status – some mixture of:

• “My best guess, based on some theory, practice and observations. But very much _not_ battle-tested”
• but also, “poetry that’s designed to get an idea across that isn’t necessarily precisely accurate", intended to get across the generators for my current worldview.
• Was waiting to post this until I resolved some disagreements that seemed upstream of it, but I think it'll be awhile before that happens. idk. YOLO.

tl;dr:

People are not automatically robust agents, and neither are organizations.

An organization can become an agent (probably?) but only if it’s built right. Your default assumption should probably be that a given organization is not an agent (and therefore may not be able to credibly make certain kinds of commitments).

Your default assumption, if you’re building an organization, should probably be that it will not be an agent (and will have some pathologies common to organizations).

If you try on purpose to make it an agent, have good principles, etc…

...well, your organization probably still won’t be an agent, and some of those principles might get co-opted by adversarial processes. But I think it’s possible for an organization to at least be better at robust agency (and, also better at being “good”, or “human value aligned”, or at least “aligned with the values of the person who founded it.”)

For a few years I’ve been crystallizing what it means to be a robust agent, by which I mean: “Reliably performing well, even if the environment around you changes. Have good policies. Have good meta policies. Be able to interface well with people who might have a wide variety of strategies, some of who might be malicious or confused.”

Becoming a robust person

People are not born automatically strategic, nor are they born an “agent.”

If you want robust agency, you have to cultivate it on purpose.

I have a friend who solves a lot of problems using the multi-agent paradigm. He spends a lot of effort integrating and empowering his sub-agents. He treats them like adults, makes sure they understand each other and trust each other. He makes sure each of them have accurate beliefs, and he tries to empower each of them as much as possible so they have no need to compete.

This… doesn’t actually work for me.

I’ve tried things like internal double crux or internal family systems, and so far, it’s just produced a confused “meh.” Insofar as “sub-agents” is a workable framework, I still have a pretty adversarial relationship with myself. (When I’m having trouble sleeping or staying off facebook, instead of figuring out what needs my sub-agents have and meeting them all... I just block facebook for 16 hours a day and program my computer to turn itself off every hour of the night starting at 11pm)

I'm tempted to write off my friend's claims as weird-posthoc-narrative. But this friend is among the more impressive people I know, and consistently has good reasons for things that initially sound weird to me. (This shouldn't be strong evidence to you, but it's enough evidence for me personally to take it seriously)

I once asked him “so… how do you even get your sub-agents to say anything to each other? I can’t tell if I have sub-agents or not but if I do they sure seem incoherent. Have you always had coherent sub-agents?”

And he said (paraphrased by me), something like:

“You know how when you’re a baby, you’re a flailing incoherent mess. And then you become, like, a four year old and you can sort of communicate but you can’t keep promises or figure things out very well. And then you’re a teenager and… maybe you’re a reasonable person, but maybe you’re still angry and moody and think you know everything even though you’re like fourteen-year-old and kinda insufferable? "But... eventually you become an actual person who can make reasonable trades, and keep contracts?“My sub-agents were like that. Initially they were incoherent like a baby. But I spent years cultivating them and teaching them and helping them grow and now they’re, like, coherent entities that have accurate beliefs and can negotiate with each other and it’s all super reasonable.”“An important element here was giving the sub-agents jobs. I looked at what Fear was doing, and one thing seemed to be “help me notice when a bad thing was going to happen to me.” And I said “Okay, Fear. This is now your official job. I will be helping you to do this. If you are doing a good job, or seem to be making mistakes, I will be giving you feedback about that.”

This… was an interesting outlook.

The jury’s still out on whether sub-agents are a useful framework for me. But this still fit into an interesting meta-framework.

Subagents or no, people don’t stop growing as agents when they become adults – there’s more to learn. I’ve worked over the past few years to improve my ability to think, and have good policies that defend my values while interfacing better with potential allies and enemies and confused bystanders.

I still have a lot more to go.

Becoming a robust organization

People are not automatically robust agents.

Neither are organizations.

Whether or not sub-agents are a valid frame for humans (or for particular humans), they seem like a pretty valid lens to examine organizations through.

An organization is born without a brain, and without a soul, and it will not have either unless you proactively build it one. And, I suspect, you are limited in your ability to build it one by the degree of soul and brain that you have cultivated in yourself. (Where “you” is “whoever is building the organization”, which might be one founder or multiple co-founders)

Vignettes of Organizational Coherence

Epistemic Status: Somewhat poetry-esque. These vignettes from different organizations paint a picture more than they spell out an explicit argument. But I hope it helps express the overall worldview I currently hold.

Holding off on Hiring

YCombinator recommends that young startups avoid hiring people as long as possible. I think there are a number of reasons for this, but one guess is that you’re ability to grow the soul of your organization weakens dramatically as it scales. It’s much harder to communicate nuanced beliefs to many-people-at-once than a few people.

The years where your organization is small, and everyone can easily talk to everyone… those are the years when you have the chance to plant the seed of agency and the spark of goodness, to ensure your organization grows into something that is aligned with your values.

The Human Alignment Problem

Ray Dalio, of Bridgewater, has a book of Principles that he endeavors to follow, and have Bridgewater follow. I disagree (or are quite skeptical about) a lot of his implementation details. But I think the meta-principle of having principles is valuable. In particular, writing things down so that you can notice when you have violated your previously stated principles seems important.

One thing he talks a lot about is “getting in sync”, which he discusses in this blog post:

For an organization to be effective, the people who make it up must be aligned on many levels—from what their shared mission is, to how they will treat each other, to a more practical picture of who will do what when to achieve their goals. Yet alignment can never be taken for granted because people are wired so differently. We all see ourselves and the world in our own unique ways, so deciding what’s true and what to do about it takes constant work.Alignment is especially important in an idea meritocracy, so at Bridgewater we try to attain alignment consciously, continually, and systematically. We call this process of finding alignment “getting in sync,” and there are two primary ways it can go wrong: cases resulting from simple misunderstandings and those stemming from fundamental disagreements. Getting in sync is the process of open-mindedly and assertively rectifying both types. Many people mistakenly believe that papering over diﬀerences is the easiest way to keep the peace. They couldn’t be more wrong. By avoiding conﬂicts one avoids resolving diﬀerences. People who suppress minor conﬂicts tend to have much bigger conﬂicts later on [...]While it is straightforward to have a meritocracy in activities in which there is clarity of relative abilities (because the results speak for themselves such as in sports, where the fastest runner wins the race), it is much harder in a creative environment (where diﬀerent points of view about what’s best have to be resolved). If they’re not, the process of sorting through disagreements and knowing who has the authority to decide quickly becomes chaotic. Sometimes people get angry or stuck; a conversation can easily wind up with two or more people spinning unproductively and unable to reach agreement on what to do.For these reasons, speciﬁc processes and procedures must be followed. Every party to the discussion must understand who has what rights and which procedures should be followed to move toward resolution. (We’ve also developed tools for helping do this). And everyone must understand the most fundamental principle for getting in sync, which is that people must be open-minded and assertive at the same time.The Treacherous Turn

This particular description about the treacherous turn (typically as applied to AI, but in this case using the example of a human) feels relevant:

To master lying, a child should:1. Possess the necessary cognitive abilities to lie (for instance, by being able to say words or sentences).2. Understand that humans can (deliberately) say falsehoods about the world or their beliefs.3. Practice lying, allowing himself/herself to be punished if caught.If language acquisition flourishes when children are aged 15-18 months, the proportion of them who lie (about peeking in a psychology study) goes from 30% at age 2, to 50% of three-year olds, eventually reaching 80% at eight. Most importantly, they get better as they get older, going from blatant lies to pretending to be making reasonable/honest guesses. There is therefore a gap between the moment children could (in theory) lie (18 months) and the moment they can effectively lie and use this technique to their own advantage (8 years old). During this gap, parents can correct the kid's moral values through education.

I’m not sure the metaphor quite holds. But it seems plausible that if you want an organization where individuals, teams and departments don’t lie (whether blatantly and maliciously, or through ‘honest goodhart-esque mistakes’, or through something like Benquo’s 4-level-simulacrum concept), you have some window in which you can try to install a robust system of honesty, honor and integrity, before the system becomes too powerful to shape.

Sometimes bureaucracy is successfully protecting a thing, and that’s good

Samo's How to Use Bureaucracies matched my experience watching bureaucracies form. I’ve seen bureaucracies form that looked reasonably formed-on-purpose-by-a-competent-person, and I’ve seen glimpses of ones that looked sort of cobbled together like spaghetti towers.

An interesting viewpoint I’ve heard recently is “usually when people are complaining that Bureaucracies don’t have souls, I think they’re just mad that the bureaucracy didn’t give them the resources they wanted. And the bureaucracy was specifically designed to stop from people like them from exploiting it.

“Academic bureaucracies, say, have a particular goal of educating people and doing research. If you come to them with a plan that will educate people or improve research, they will usually give you want you want. If you come to them trying to get weird special exceptions or faculties for saving the world or whatever, they’ll be like ‘um, our job is not to save the world, it is to educate people and do research. If we gave resources to every person with a pet cause, we’d fall apart immediately.'”“Likewise, if they impose a weird rule on you, it’s probably because in the past sometime fucked up in some way relating to that rule. And dealing with the fallout was really annoying, and they decided they didn’t want to have to deal with that fallout ever again. Sorry that you think you’re a good exception or the rule is stupid – part of the point of policies is to abstract away certain things so they can’t bother you and you can focus on what matters.”

I’m not sure how often this is actually true and how often it’s just a convenient story (bureaucracies do seem to be built out of spaghetti towers). But it seems plausible in at least some cases. And it seems noteworthy that “having a soul” might be compatible with “include leviathanic institutions that don’t seem to care about you as a person.”

Sabotaging the Nazis

On the flipside...

LW user Lionhearted notes in Explicit and Implicit communication that during World War II, some allies went to infiltrate the Nazis and gum up the works. They received explicit instructions like:

“(11) General Interference with Organizations and Production [...](1) Insist on doing everything through “channels.” Never permit short-cuts to be taken in order to expedite decisions.(2) Make “speeches.” Talk as frequently as possible and at great length. Illustrate your “points” by long anecdotes and accounts of personal experiences. Never hesitate to make a few appropriate “patriotic” comments.(3) When possible, refer all matters to committees, for “further study and consideration.” Attempt to make the committees as large as possible — never less than five. [...](5) Haggle over precise wordings of communications, minutes, resolutions.(6) Refer back to matters decided upon at the last meeting and attempt to re-open the question of the advisability of that decision.(7) Advocate “caution.” Be “reasonable” and urge your fellow-conferees to be “reasonable” and avoid haste which might result in embarrassments or difficulties later on.(8) Be worried about the propriety of any decision — raise the question of whether such action as is contemplated lies within the jurisdiction of the group or whether it might conflict with the policy of some higher echelon.”

And... well, this all sure sounds like the pathologies I normally associate with bureaucracy. This sort of thing seems to happen by default, as an organization scales.

There's also Scott's IRB Nightmare.

Organizations have to make decisions and keep promises.

Why can’t you just have individual agents within an organization? Why does it matter that the organization-as-a-whole be an agent?

If you can’t make “real” decisions and keep commitments, you will be limited in your ability to engage in certain strategies, in some cases unable to engage in mutually beneficial trade.

Organizations control resources that are often beyond the control of a single person, and involve complicated decision making procedures. Sometimes the procedure is a legible, principled process. Sometimes a few key people in the room-where-it-happens hash things out, opaquely. Sometimes it’s a legible-but-spaghetti-tower bureaucracy.

Any of these can be fine. But it’s usually still something beyond the sum-of-the-individual people involved.

Sometimes nobody has any power – everyone requires too many checks from too many other people and nothing gets done on purpose.

Sometimes you talk to the head of the org, and maybe you even trust the head of the org, and they say the org will do a thing, but somehow the org doesn’t end up doing the thing.

Sometimes, you can talk to each individual person at the org and they all agree Decision X would be best, but they’re all afraid to speak up because there isn’t common knowledge that they agree with Decision X. Or, they do all agree and know it, but they can’t say it publicly because The Public doesn’t understand Decision X.

So Decision X doesn’t get made.

Sometimes you talk to each individual person and they each individually agree that Decision X is good, and you talk to the entire group and the entire group seems to agree that Decision X would be good, but… somehow Decision X doesn’t get done.

I think it makes sense for bureaucracies to exist sometimes, and to have the explicit purpose of preventing people from exploiting things too easily. But, it’s still useful for some part of the institution to be able to make decisions and commitments that weren’t part of explicitly-laid-out bureaucracy chain.

Porous movements aren’t and can’t be agents

I think that agency requires a membrane, something keeps particular people in and out, such that you have any deliberate culture, principles or decision making at all.

Relatedly, I think you need a membrane for Stag Hunts to work – if any rando can blunder into the formation at the last moment, there’s no way you can catch a stag.

Organizations have fairly strong membranes, and sometimes informal community institutions can as well. But this is relatively rare.

So while I’m disappointed sometimes when particular individuals and organizations don’t live up to the ideals I think they were trying for… I don’t think it makes much sense to hold most “movements” to the ideal of agency. Movements are too chaotic, too hard to police, too easy to show up in and start shouting and taking up attention.

Instead, instead, I think of movements as a place where a lot of people with similar ideals are clustered together. This makes it easier to find recruit people into organizations that do have membranes and can have principles.

Narrative control and contracts, as alternative coordination mechanisms

Another friend who ran an organization once remarked (paraphrased)

“It seemed like the organization’s main coordination mechanism was a particular narrative that people rallied around. When I was in charge, I felt like it was my job to uphold that narrative, even when the narrative got epistemically dicey. This felt really bad for my soul, and eventually I stopped being in charge.”“I’m not sure what to do about this problem – organizations need some kind of coordination mechanism. I think a potential solution might be to make central element of your company culture ‘upholding contracts.’ Maybe you don’t all share the same vision for the company, but you can make concrete trades. Some of those trades are “I will do X and you will pay me dollars”, and some might be between employees, like “I will work enthusiastically on this aspect of the company for 2 months if you work enthusiastically on that aspect of it.”

This seems plausible to me. But importantly, I don’t think you get “uphold contracts” as a virtue for free. If you want your employees to be able to do it reliably, you need mechanisms to train and reinforce that. (I think if you recruit from some homogenous cultures it might come more automatically, but it’s not my default experience)

Integrity and Accountability

Habryka recently wrote about Integrity and Accountability, and it seemed useful to just quote the summary here:

One lens to view integrity through is as an advanced form of honesty – “acting in accordance with your stated beliefs.”— To improve integrity, you can either try to bring your actions in line with your stated beliefs, or your stated beliefs in line with your actions, or reworking both at the same time. These options all have failure modes, but potential benefits.— People with power sometimes have incentives that systematically warp their ability to form accurate beliefs, and (correspondingly) to act with integrity.An important tool for maintaining integrity (in general, and in particular as you gain power) is to carefully think about what social environment and incentive structures you want for yourself.Choose carefully who, and how many people, you are accountable to:— Too many people, and you are limited in the complexity of the beliefs and actions that you can justify.— Too few people, too similar to you, and you won’t have enough opportunities for people to notice and point out what you’re doing wrong. You may also not end up with a strong enough coalition aligned with your principles to accomplish your goals.Open Problems in Robust Group Agency

Exercises for the reader, and for me:

1. How do you make sure your group has any kind of agency at all, let alone be ‘value-aligned’

2. How do you choose people to be accountable to? What if you’re trying to do something really hard, and there seem to be few or zero people who you trust enough to be accountable to?

3. It seems like the last cluster of people who tried to solve accountability created committees and boards and bureaucracies, and… I dunno, maybe that stuff works fine if you do it right. But it seems easy to become dysfunctional in particular ways. What’s up with that?

3. What “rabbit” strategies are available, within and without organizations, that are self-reinforcing in the near term, that can help build trust, accountability, and robust agency?

4. What “stag” strategies could you successfully execute on if you had a small group of people working hard together?

4b. How can you get a small group of dedicated, aligned people?

5. How can people maintain accurate beliefs in the face of groupthink?

6. How can any of this scale?

Discuss

### Thoughts on the 5-10 Problem

18 июля, 2019 - 21:59
Published on July 18, 2019 6:56 PM UTC

5 dollars is better than 10 dollars

The 5-10 Problem is a strange issue in which an agent reasoning about itself makes an obviously wrong choice.

Our agent faces a truly harrowing choice: it must decide between taking $5 (utility 5) or$10 (utility 10).

How will our agent solve this dilemna? First, it will spend some time looking for a proof that taking $5 is better than taking$10. If it can find one, it will take the $5. Otherwise, it will take the$10.

Fair enough, you think. Surely the agent will concede that it can't prove taking $5 is better than taking$10. Then, it will do the sensible thing and take the $10, right? Wrong. Our agent finds the following the following proof that taking$5 is better:

Let's go over the proof.

Line 1: Taking $5 gives you$5.

Line 2: If F is true, then ~F->x is true for any x.

Line 3: If you find a proof that taking $5 gives you$5 and take $10 gives you$0, you'd take the $5. Line 4: Combine the three previous lines Line 5: Löb's Theorem Line 6: Knowing that taking$5 gives you $5 and taking$10 gives you $0, you happily take the$5.

Simplified Example

To understand what went wrong, we'll consider a simpler example. Suppose you have a choice between drinking coffee (utility 1) and killing yourself (utility -100).

You decide to use the following algorithm: "if I can prove that I will kill myself, then I'll kill myself. Otherwise, I'll drink coffee".

And because a proof that you'll kill yourself, implies that you'll kill yourself, by Lob's Theorem, you will kill yourself.

Here, it is easier to see what went wrong-proving that you'll kill yourself is not a good reason to kill yourself.

This is hidden in the original 5-10 problem. The first conditional is equivalent to "if I can prove I will take $5, then I'll take$5".

Hopefully, it's now more clear what went wrong. How can we fix it?

Solution?

I once saw a comment suggesting that the agent instead reason about how a similar agent would act (I can't find it anymore, sorry). However, this notion was not formalized. I propose the following formalization:

Discuss

### Prereq: Question Substitution

18 июля, 2019 - 20:35
Published on July 18, 2019 5:35 PM UTC

A quote from a 2002 paper by Kahneman and Frederick referencing some of Kahneman's earlier work with Tversky:

Early research on the representativeness and availability heuristics was guided by a simple and general hypothesis: when confronted with a difficult question people often answer an easier one instead, usually without being aware of the substitution.

It's an interesting hypothesis, and even without looking at studies it seems plausible. I can think of conversations where I've been frustrated by the fact that the other person can't seem to actually answer the question I've asked them and keeps wiggling around. But it makes me wonder.

How would you tell the difference between A:

Question is substituted for an easier one.
Brain replies with answer to easier question

and B:

Brain replies using a heuristic that it applies to this sort of question.

(thanks to this comment for helping me notice the discrepancy)

Hmmm, it seems like they could easily produce the same answer. Does question substitution explain any more than "the brain uses heuristics"?

Yes and no. I read the first few pages of the paper, and pretty quickly noticed that Kahneman doesn't seem to think that a literal question substitution is happening. Most of the paper explores the question of "what governs what heuristic gets applied when?" and in other phrasings Kahneman states "[people reply to questions] as if she had been asked [other question]."

Despite the fact that literal question substitution might not happen that much, I can still get useful information by using the substitution frame. If I'm investigating whether or not I used a heuristic I wasn't aware of, I can ask myself: "Was I trying to answer an easier question? What are similar but simpler questions related to the one I was trying to answer? Are there any questions I know I'm good at answering that I might have substituted for the original one?"

You might even say that question substitution is a great... heuristic for figuring out if you were using a heuristic, in lieu of a more detailed model of how and when your brain picks heuristics. Yes, by all means read the rest of the paper and launch a quest into understanding in detail how the mind works. And while you're working on that, feel free to use the idea of question substitution to help you explore.

One thing that question substitution helps me notice is my tendency to "have a hammer and go looking for nails." When doing a substitution, what sort of other questions might you want to substitute in? Questions you're good at answering! This will be relevant later on when when exploring ways you can accidentally get stuck in never ending arguments.

Isn't this just...

No. It's not. To be more explicit about what I was saying in cognitive fusion, I frequently see people take the following frame:

Sometimes, for no reason people make dumb mistakes, which they'd better fix as soon as I point them out.
"Well gee, there's got to be some reason people make these mistakes."
"Yeah, it's because they're dumb/irrational/unenlightened."

At best, telling someone not to make "dumb mistakes" let's them know how you are going to judge them. At worst, it asserts a world where "dumb mistakes" are this magical fundamental thing your brain spits out sometimes, and that you fix it by turning off the "dumb mistake" switch in your head. We all know where that is, right? Cool, just checking.

Thinking about question substitution orients me to the process of my brain swapping in a heuristic, instead of becoming fixated on the heuristic itself and how positively idiotic it is and how I can't believe anyone could possibly be dumb enough to use such a heuristic and I'm so glad that I don't use any heuristics that stupid...

And so it goes.

Fusion, substitution, and your journey into the mind

I've been ignoring the second part of substitution, the part where you don't notice that you did a substitution. Hmmm, not noticing when something that feels like "just giving an answer" is actually composed of a multi step heuristic selection process. If I squint, this sort of looks like fusion. A very light sort of fusion, depending on how readily I go "Oh yeah, oops" when the substitution is pointed out.

"Wait, you seem to be diluting the meaning of fusion to refer to any sort of lack of awareness of what your mind is doing!"

Yeah, I'm definitely using fusion pretty broadly. I'm proposing that you can use fusion and substitution as two general lenses to explore how you're mind actually works. What is your mind doing, and how much are you aware of what it is doing? If you dig into ACT, you'll find that cognitive fusion is a richer concept with more backing than what I'm describing. If you read the paper I linked at he beginning, you'll find Kahneman and Frederick digging into all sorts of interesting mechanisms that govern how and when heuristics get applied.

I'm proposing thinking in terms of fusion and substitution as a first step out of "dumb mistakes" thinking. "I don't like the fact that my future is an empty hopeless void". Maybe I'm "just being dumb" and I should "get over it." Or maybe I substituted "Will there be anything good in my future?" with "Is there anything good in my life right now?" (because predicting the future is hard) and then fused to that (because it's a super emotionally charged low level thought and I've never trained in defusion). They both point to the same problem. Which one is easier to solve?

Discuss

### The National Security Commission on Artificial Intelligence Wants You (to submit essays and articles on the future of government AI policy)

18 июля, 2019 - 20:21

### Laplace Approximation

18 июля, 2019 - 18:23
Published on July 18, 2019 3:23 PM UTC

The last couple posts compared some specific models for 20000 rolls of a die. This post will step back, and talk about more general theory for Bayesian model comparison.

P[data|model]=∫θP[data|θ]dP[θ]

which will be a hairy high-dimensional integral.

Some special model structures allow us to simplify the problem, typically by factoring the integral into a product of one-dimensional integrals. But in general, we need some method for approximating these integrals.

The two most common approximation methods used in practice are Laplace approximation around the maximum-likelihood point, and MCMC (see e.g. here for application of MCMC to Bayes factors). We'll mainly talk about Laplace approximation here - in practice MCMC mostly works well in the same cases, assuming the unobserved parameters are continuous.

Laplace Approximation

Here's the idea of Laplace approximation. First, posterior distributions tend to be very pointy. This is mainly because independent probabilities multiply, so probabilities tend to scale exponentially with the number of data points. Think of the probabilities we calculated in the last two posts, with values like 10−70 or 10−20 - that's the typical case. If we're integrating over a function with values like that, we can basically just pay attention to the region around the highest value - other regions will have exponentially small weight.

Laplace' trick is to use a second-order approximation within that high-valued region. Specifically, since probabilities naturally live on a log scale, we'll take a second order-approximation of the log likelihood around its maximum point. Thus:

∫θelnP[data|θ]dP[θ]≈∫θelnP[data|θmax]+12(θ−θmax)T(d2lnPdθ2|θmax)(θ−θmax)dP[θ]

If we assume that the prior dP[θ] is uniform (i.e. dP[θ]=dθ), then this looks like a normal distribution on θ with mean θmax and variance given by the inverse Hessian matrix of the log-likelihood. (It turns out that, even for non-uniform dP[θ], we can just transform θ so that the prior looks uniform near θmax, and transform it back when we're done.) The result:

∫θelnP[data|θ]dP[θ]≈P[data|θmax]p[θmax](2π)k2det(−d2lnPdθ2|θmax)−12

Let's walk through each of those pieces:

• P[data|θmax] is the usual maximum likelihood: the largest probability assigned to the data by any particular value of θ.
• p[θmax] is the prior probability density of the maximum-likelihood θ point.
• (2π)k2 is that annoying constant factor which shows up whenever we deal with normal distributions; k is the dimension of θ.
• det(−d2lnPdθ2|θmax) is the determinant of the "Fisher information matrix"; it quantifies how wide or skinny the peak is.

A bit more detail on that last piece: intuitively, each eigenvalue of the Fisher information matrix tells us the approximate width of the peak in a particular direction. Since the matrix is the inverse variance (in one dimension 1σ2) of our approximate normal distribution, and "width" of the peak of a normal distribution corresponds to the standard deviation σ, we use an inverse square root (i.e. the power of −12) to extract a width from each eigenvalue. Then, to find how much volume the peak covers, we multiply together the widths along each direction - thus the determinant, which is the product of eigenvalues.

Why do we need eigenvalues? The diagram above shows the general idea: for the function shown, the two arrows would be eigenvectors of the Hessian d2lnPdθ2 at the peak. Under a second-order approximation, these are principal axes of the function's level sets (the ellipses in the diagram). They are the natural directions along which to measure the width. The eigenvalue associated with each eigenvector tells us the width, and then taking their product (via the determinant) gives a volume. In the picture above, the determinant would be proportional to the volume of any of the ellipses.

Altogether, then, the Laplace approximation takes the height of the peak (i.e. P[data|θmax]p[θmax]) and multiplies by the volume of θ-space which the peak occupies, based on a second-order approximation of the likelihood around its peak.

Laplace Complexity Penalty

The Laplace approximation contains our first example of an explicit complexity penalty.

The idea of a complexity penalty is that we first find the maximum log likelihood lnP[data|θmax], maybe add a term for our θ-prior lnp[θmax], and that's the "score" of our model. But more general models, with more free parameters, will always score higher, leading to overfit. To counterbalance that, we calculate some numerical penalty which is larger for more complex models (i.e. those with more free parameters) and subtract that penalty from the raw score.

In the case of Laplace approximation, a natural complexity penalty drops out as soon as we take the log of the approximation formula:

lnP[data|model]≈lnP[data|θmax]+lnp[θmax]+k2ln(2π)−12lndet(−d2lnPdθ2|θmax)

The last two terms are the complexity penalty. As we saw above, they give the (log) volume of the likelihood peak in θ-space. The wider the peak, the larger the chunk of θ-space which actually predicts the observed data.

There are two main problems with this complexity penalty:

• First, there's the usual issues with approximating a posterior distribution by looking at a single point. Multimodal distributions are a problem, insufficiently-pointy distributions are a problem. These problems apply to basically any complexity penalty method.
• Second, although the log determinant of the Hessian can be computed via backpropagation and linear algebra, that computation takes O(k3). That's a lot better than the exponential time required for high-dimensional integrals, but still too slow to be practical for large-scale models with millions of parameters.

Historically, a third issue was the math/coding work involved in calculating a Hessian, but modern backprop tools like Tensorflow or autograd make that pretty easy; I expect in the next few years we'll see a lot more people using a Laplace-based complexity penalty directly. The O(k3) runtime remains a serious problem for large-scale models, however, and that problem is unlikely to be solved any time soon: a linear-time method for computing the Hessian log determinant would yield an O(n2) matrix multiplication algorithm.

Discuss

### Normalising utility as willingness to pay

18 июля, 2019 - 14:44
Published on July 18, 2019 11:44 AM UTC

I've thought of a framework that puts most of the methods of interteoretic utility normalisation and bargaining on the same footing. See this first post for a reminder of the different types of utility function normalisation.

Most of the normalisation techniques can be conceived of as a game with two outcomes, and each player can pay a certain amount of their utility to flip from one one outcome to another. Then we can use the maximal amount of utility they are willing to pay, as the common measuring stick for normalisation.

Consider for example the min-max normalisation: this assigns utility 0 to the expected utility if the agent makes the worst possible decisions, and 1 if they make the best possible ones.

So, if your utility function is u, the question is: how much utility would you be willing to pay to prevent your nemesis (a −u maximiser) from controlling the decision process, and let you take it over instead? Dividing u by that amount[1] will give you the min-max normalisation (up to the addition of a constant).

Now consider the mean-max normalisation. For this, the game is as follows: how much would you be willing to pay to prevent a policy from choosing randomly amongst the outcomes ("mean"), and let you take over the decision process instead?

Conversely, the mean min-mean normalisation asks how much you would be willing to pay to prevent your nemesis from controlling the decision process, and shifting to a random process instead.

The mean difference method is a bit different: here, two outcomes are chosen at random, and you are asked now much you are willing to pay to shift from the worst outcome to the best. The expectation of that amount is used for normalisation.

The mutual Worth bargaining solution has a similar interpretation: how much would you be willing to pay to move from the default option, to one where you controlled all decisions?

A few normalisations don't seem to fit into the this framework, most especially those that depend on the square of the utility, such as variance normalisation or the Nash Bargaining solution. The Kalai–Smorodinsky bargaining solution uses a similar normalisation as the mutual worth bargaining solution, but chooses the outcome differently: if the default point is at the origin, it will pick the point (x,x) with largest x.

1. This, of course, would incentivise you to lie - but that problem is unavoidable in bargaining anyway. ↩︎

Discuss

### Dialogue on Appeals to Consequences

18 июля, 2019 - 05:34
https://s0.wp.com/i/blank.jpg

### Why it feels like everything is a trade-off

18 июля, 2019 - 04:40
Published on July 18, 2019 1:33 AM UTC

Epistemic status: A cute insight that explains why it might feel like you always have to make sacrifices along one metric to get improvements along another. Seems tautological once you understand it. Might be obvious to everyone.

Meta-epistemic status or something: My first post. Testing the waters.

I'm a programmer. I'm also a design prude. I'm also lazy. This all means that I spend a lot of my time chasing some different metrics in my code:

1) How easy it is to read.

2) How long it takes to run.

3) How long it takes to write.

4) ...

These metrics are often at odds with one another. Just the other day I had to make a trade-off involving a function I'd written to evaluate a polynomial at a given point. Originally, it was written in a way that I felt was self-explanatory: it looped over the derivatives-at-zero of the polynomial, which were passed in as a list, and summed up the appropriate multiples of powers of x — a Taylor sum. Pseudo-code:

def apply_polynomial( deriv, x ):
sum = 0
for i from 0 to length( deriv ):
sum += deriv[i] * pow(x, i) / factorial(i)
return sum

It turned out that this function was a significant bottleneck in the execution time of my program: about 20% of it was spent inside this function. I was reasonably sure that the pow and factorial functions were the issue. I also knew that this function would only ever be called with cubics and lower-degree polynomials. So I re-wrote the code as follows:

def apply_cubic( deriv, x ):
sum = 0
len = length( deriv )

if len > 0:
sum += deriv[0]
if len > 1:
sum += deriv[1] * x
if len > 2:
square = x * x
sum += deriv[2] * square / 2
if len > 3:
cube = square * x
sum += deriv[3] * cube / 6

return sum

Sure enough, this improved the runtime significantly — by nearly the whole 20% that had been being spent inside this function. But notice that the code no longer contains the elements that define a Taylor sum: the loop is gone, and the factorials (0!, 1!, 2!, 3!) have been replaced with their values (1, 1, 2, 6). It also isn't obvious why the length comparisons stop at 3. The code no longer explains itself, and must be commented to be understood. Readability has been sacrificed on the altar of efficiency.

Question

Why am I cursed so? Why can't these metrics go hand-in-hand? And in general, why am I always doing this sort of thing? Sacrificing flavor for nutrition in the cafeteria, sacrificing walking-speed for politeness on a crowded sidewalk? Why are my goals so often set against one another?

Suppose you're faced with a problem to solve. You have two solutions in mind (Solution A & Solution B), and you have two metrics (Metric 1 & Metric 2) by which to judge a solution. A trade-off occurs when the solution that scores better along Metric 1 scores worse along Metric 2:

For example:

Of course, there need not be only two possible solutions. Maybe I'm willing to spend two hours working on improving this function, and depending on what I focus on, I could achieve any of the following balances:

And — argh! The correlation is negative! Why!

Well, there's a reason, and the reason is that this isn't the full picture. This is:

See, there are a whole bunch of ways to write the code that are neither as efficient nor as readable as one of the filled-in circles on the perimeter. But I would never choose any of those ways, for obvious reasons. And if a new solution occurs to me that beats out some of my old solutions along both metrics...

...then this new solution would replace all the solutions strictly worse than it, which in turn would become part of the mass that resides below the curve:

No matter how many new solutions are introduced into the mix, and no matter by how far they out-perform the old solutions, the outer frontier of non-dominated solutions must have negative slope everywhere. A step to the right along this line must be accompanied by a step downward, because if it isn't, then the solution you just stepped off of is dominated by the one you just stepped onto, so the former wasn't on the line.

It doesn't take any math to generalize this result to situations where you have more than two metrics. Any solution that is dominated by another solution will be thrown out, so the options you end up considering form a set where no element dominates another, a.k.a. one where a gain along one metric entails a loss along at least one other, a.k.a. a trade-off. Throwing solutions out is easy (and is sometimes done subconsciously when you're working in your domain of expertise, or done by other people before a decision gets to you), but weighing the remaining options usually takes some consideration. So, our subjective experience is that we spend most of our time and energy thinking about trade-offs. Q.E.D.

Discuss

### Prereq: Cognitive Fusion

17 июля, 2019 - 22:04
Published on July 17, 2019 7:04 PM UTC

In a post by Kaj Sotala, he introduces the very useful idea of cognitive fusion.

Cognitive fusion is a term from Acceptance and Commitment Therapy (ACT), which refers to a person “fusing together” with the content of a thought or emotion, so that the content is experienced as an objective fact about the world rather than as a mental construct. The most obvious example of this might be if you get really upset with someone else and become convinced that something was all their fault (even if you had actually done something blameworthy too).
In this example, your anger isn’t letting you see clearly, and you can’t step back from your anger to question it, because you have become “fused together” with it and experience everything in terms of the anger’s internal logic.

You can become fused to an emotion, a voice in your head, a political view, and experience it to "just be true". I see this as a similar sort of fusion I hear musician talk about, where after years of practice their instrument begin to feel like a part of their body. They aren't "using their index finger to press the black note on a piano" they are "just playing G". This is analogous to being so caught up in your own anger that your partner is "just wrong and terrible" as opposed to "it sorta looks like you intentionally did something to annoy me and I'm worried about if you'll do this again in the future." (or whatever the actual case is)

Sometimes I think of there being a general fusion process where the brain collapses levels of inference. All of the steps that go into a given physical motion or thought process get compressed into a single dot. The thought process will be experienced as "just true" and the physical motion will be experience as an atomic action available to you. Sometimes you can "uncompress" the chain, and sometimes you can't.

Problems can arise when you fuse to a thought or emotion that doesn't have an accurate view of the world, and you unknowingly take it's broken map as the territory.

Isn't this just "Don't make assumptions"?

Not quite, though it is similar. Assumptions don't really capture the more general fusing process that you can also see with physical movement. "I can't believe that you just assumed you start off on your left foot when making a layup!" Nah, doesn't feel right. But the main reason I prefer to talk in terms of fusion is that "fusion" makes me focus on the process of attaching to something, while "assumptions" makes me focus on the object being attached to.

It's easier to see this difference when the though being fused to (or the claim being assumed) is "obviously" wrong, or at least obvious to one who isn't fused to it. The assumption frame makes me feel like my work is done when I find the other persons "dumb" assumption. Point it out with a pithy "Checkmate [outgroup]" and move on. The fusion frame leads me to ask "How did they get fused to this in the first place? How might I help them defuse from it?" By focusing on the process of attachment (fusion) I can appreciate how common it is to fuse to something and how hard it can be to defuse. When I focus on the object of attachment (assumption) I'm mostly thinking about just how stupid it is and how I can't believe that anyone would be dumb enough to fall for this, and I most certainly don't believe anything that stupid....

And so it goes. Thinking of some behavior as a "dumb mistake" makes you more likely to not notice when you engage in it. Some thoughts take moments to defuse from. Others take a lifetime. Sometimes you fuse to things, and you'd be wise to learn how it works rather than to ridicule it.

As you may have guessed, later parts of this sequence will talk about what can happen when you fuse with language. For now, just remember what fusion is, and treat it with respect.

"Modern man can't see God because he doesn't look low enough."

-- Carl Jung

Discuss

### Why did we wait so long for the bicycle?

17 июля, 2019 - 21:45
Published on July 17, 2019 6:45 PM UTC

h/t alyssavance

(Continue reading. 2,184 words and lots of great bicycle pictures.)

This post is a single piece from Jason Crawford's project, The Roots of Progress, aptly named, to understand the nature and causes of human progress. I haven't thought deeply enough to check his research, but it's a fascinating project. This essay examines a specific piece of technology, but the case study is used to develop and support models of what it takes for progress to occur.

Discuss

### 1st Athena Rationality Workshop - Retrospective

17 июля, 2019 - 19:51
Published on July 17, 2019 4:51 PM UTC

During June 7-10 we ran the first Athena Rationality Workshop at the EA Hotel. The workshop taught the Ease process, which is a mental self debugging framework developed by Matt Goldenberg. The Ease framework incorporates various things he learned and developed during his 5 years of working as a personal coach.

The Ease process is a step by step method going through

• Awareness - noticing you thoughts, feelings and other mental objects
• Introspection - finding the root cause of what is going on in your mind
• Acceptance - accepting that you are who you are in the present and the choices you have done in the past
• Alignment - creating peace between your internal parts that are involved in the issue you are trying to fix
• Ecology - checking that whatever shift you made in you mind during the alignment step, does not create new problems elsewhere
• Creation - building the attitude, habits and external support that you need to keep on track with your new solution
• Integration - deeply integrate and reinforce new habits and attitude into your mind

Most of the material was taught by Matt. Ryan Thomas, Deni Pop, Toon Alfrink and Linda Linsefors (me) gave one or two lectures each and, to different extents, helped participants during exercises.

Overall the workshop went really well. We got several pieces of positive feedback during the workshop, and from the feedback form as well, although there is room for improvement (see further down). On average the participants estimated the time spent at the workshop to 9.5 times more productive compared to what they would have done otherwise, with answers ranging from 0.5 to 50.

"I was very impressed by the quality of the workshop; to be able to string together so many disparate theories, tools, and techniques into a single coherent framework, is nothing short of genius. Brilliant work!" - Olivier Maas (participant)

Who we are

Matt Goldenberg has been working intensely on self-improvement for the past 15 years, was a professional coach for 5 years and has been running applied rationality group workshops for the past 18 months.

Linda Linsefors organizes workshops. She gets the job done.

Toon Alfrink firmly decided to turn around his mental health issues once and for all. Somehow, this time, it actually worked. He visited a Zen Monastery for 3 months and hasn’t been depressed ever since. His productivity benefited just as much, going from struggling at half speed in uni to running an AI Safety startup, the LW Netherlands community, and getting good grades in uni all at once. To the extent that he has an idea how the hell all of this happened, he would like to teach you.

Denisa Pop was a licensed cognitive-behavioral therapist for 5 years and has a PhD in positive human-animal interaction. Now she uses her previous knowledge to improve human-human interaction, doing research in rational compassion and being a Community Manager at the EA hotel.

Ryan Thomas is a student of mindfulness and cognitive psychology. He’s spent the last several years traveling in order to learn and practice techniques for resolving internal conflict and solidifying self alignment.

Things that could have gone better

The first Athena Rationality Workshop was definitely rushed in several ways. We started planning the workshop just a few weeks before the event which was not enough time. The schedule was finalized only the day before the workshop.

If we had started planning earlier we would have had time to notice that the schedule was way too full and time to figure out which parts would be OK to cut out which out hurting the rest of the program. Some lessons were cut out during the workshop, but mostly the result of this mistake was that the workshop itself was rushed too. The participants did not get enough time to practice the techniques or to rest and digest what had been learned.

Another mistake that can also be traced to not having enough preparation time, is that some of us who were supposed to be around as mentors during exercises (me among others) were badly prepared. For me about two thirds of the content of the workshop was new, so I ended up spending most of the workshop just learning the stuff for myself, rather than helping others.

How to do this better next time

The reason the workshop planning was rushed is because we wanted to do it before Matt flew back to the US. Buy the time we decided to run the workshop, we could do it fast or not at all. This will not be a problem for the next workshop because we have already started the preparations.

As for Matt’s location, given the outcome of the first workshop, we are more secure in the value of the Athena Rationality Workshop, which means we can charge enough money for the event, in order to just fly Matt over when we need him.

Based on the feedback from the first Athena Rationality Workshop, we are going to cut down the content to only include the most useful and essential parts. The next workshop will be both shorter and less intense. (We are also discussing developing a more advanced week long version of the workshop, but that will take a bit longer.)

Before the next workshop, we also plan to have a few days of mentorship training, so that everyone who is mentoring during the workshop knows all the techniques, and knows what to do during the workshop.

The Next Athena Rationality Workshop

We are planning to run the next Athena Rationality Workshop some time in October or November, possibly adjacent to EA Global in London. If you want to participate, please fill in our interest form.

Online version

We are also developing an online version and are currently looking for alpha testers. The commitment would be 30 minutes a day for 7 days, each day practicing a new skill related to dealing with procrastination and akrasia. Click here for more info and signup.

Discuss

### Wolf's Dice II: What Asymmetry?

17 июля, 2019 - 18:22
Published on July 17, 2019 3:22 PM UTC

In the previous post, we looked at Rudolph Wolf's data on 20000 rolls of a pair of dice. Specifically, we looked at the data on the white die, and found that it was definitely biased. This raises an interesting question: what biases, specifically, were present? In particular, can we say anything about the physical asymmetry of the die? Jaynes addressed this exact question; we will test some of his models here.

Elongated Cube Models

Jaynes suggests that, if the die were machined, then it would be pretty easy to first cut an even square along two dimensions. But the cut in the third dimension would be more difficult; getting the length to match the other two dimensions would be tricky. Based on this, we'd expect to see an asymmetry which gives two opposite faces (1 & 6, 2 & 5, or 3 & 4) different probabilities from all the other faces.

Here's what the model looks like for the 1 & 6 pair:

• 2, 3, 4 & 5 each have the same probability 1−p4

Let's call this model1,6.

I will omit the details of calculations in this post; readers are welcome to use them as exercises. (All the integrals can be evaluated using the dirichlet-multinomial α=1 formula from the previous post.) In this case, we find

P[data|model1,6]=n!1!(n+1)!((n1+n6)!n1!n6!(12)n1+n6)((n2+...+n5)!n2!…n5!(14)n2+n3+n4+n5)≈2.2∗10−59

For the other two opposite face pairs, we get:

• 2,5: 1.4∗10−63
• 3,4: 8.5∗10−29

... sure enough, an asymmetry on the 3,4 axis goes a very long way toward explaining this data.

Recall from the previous post that the unbiased model gave a marginal likelihood P[data|model] around 10−70, and the biased model with separate probabilities for each face gave around 10−20. So based on the data, our 3,4 model is still about a billion times less probable than the full biased model (assuming comparable prior probabilities for the two models), but it's getting relatively close - probabilities naturally live on a log scale. It looks like the 3-4 asymmetry is the main asymmetry in the data, but some other smaller asymmetry must also be significant.

Just for kicks, I tried a model with a different probability for each pair of faces, again with uniform prior on the p's. That one came out to 1.7∗10−30 - somewhat worse than the 3,4 model. If you're used to traditional statistics, this may come as a surprise: how can a strictly more general model have lower marginal likelihood P[data|model]? The answer is that, in traditional statistics, we'd be looking for the unobserved parameter values p with the maximum likelihood P[data|model,p] - of course a strictly more general model will have a maximum likelihood value at least as high. But when computing P[data|model], we're integrating over the unobserved parameters p. A more general model has more ways to be wrong; unless it's capturing some important phenomenon, a smaller fraction of the parameter space will have high P[data|model,p]. We'll come back to this again later in the sequence.

Pip Asymmetry Model

Jaynes' other main suggestion was that the pips on the die are asymmetric - i.e. there's less mass near the 6 face than the 1 face, because more pips have been dug out of the 6 face.

As a first approximation to this, let's consider just the asymmetry between 1 and 6 - the pair with the highest pip difference. We'll also keep all the structure from the 3,4 model, since that seems to be the main asymmetry. Here's the model:

• 3 & 4 have the same probability p2 , as before
• 2 & 5 have the same probability 1−p4, as before
• 1 & 6 together have probability 1−p2, same as 3 and 5 together, but their individual probabilities may be different. Conditional on rolling either a 1 or 6, 1 comes up with probability p′ and 6 with probability (1−p′)
• Both p and p′ have uniform priors

The conditional parameterization for 1 & 6 is chosen to make the math clean.

Let's call this model3,4+pip. Marginal likelihood:

P[data|model3,4+pip]=n!1!(n+1)!((n3+n4)!n3!n4!(12)n3+n4)∗((n1+n2+n5+n6)!(n1+n6)!n2!n5!(12)n1+n6(14)n2+n5)((n1+n6)!1!(n1+n6+1)!)≈2.3∗10−16

... and now we have a model which solidly beats separate probabilities for each face!

(I also tried a pip model by itself, without the 3,4 asymmetry. That one wound up at 2.1∗10−70 - almost as bad as the full unbiased model.)

We can also go one step further, and assume that the pip difference also causes 2 and 5 to have slightly different probabilities. This model gives P[data|model]≈3.9∗10−17 - a bit lower than the model above, but close enough that it still gets significant posterior probability (about 3.9∗10−173.9∗10−17+2.3∗10−16=14% assuming equal priors; all the other models we've seen have near-zero posterior assuming equal priors). So based on the data, the model with just the 1-6 pip difference is a bit better, but we're not entirely sure. My guess is that a fancier model could significantly beat both of these by predicting that the effect of a pip difference scales with the number of pips, rather than just using whole separate parameters for the 1-6 and 2-5 differences. But that would get into hairier math, so I'm not going to do it here.

To recap, here's what model3,4+pip says:

• 3 and 4 have the same probability, but that probability may be different from everything else
• 2 and 5 have the same probability, and 1 and 6 together have the same probability as 2 and 5, but 1 and 6 have different probabilities.

That's it; just two "free parameters". Note that the full biased model, with different probabilities for each face, is strictly more general than this - any face probabilities p which are compatible with model3,4+pip are also compatible with the full biased model. But the full biased model is compatible with any face probabilities p; model3,4+pip is not compatible with all possible p's. So if we see data which matches the p's compatible with model3,4+pip, then that must push up our posterior for model3,4+pip relative to the full unbiased model - model3,4+pip makes a stronger prediction, so it gets more credit when it's right. The result: less flexible models which are consistent with the data will get higher posterior probability. The "complexity penalty" is not explicit, but implicit: it's just a natural consequence of conservation of expected evidence.

Next post we'll talk about approximation methods for hairy integrals, and then we'll connect all this to some common methods for scoring models.

Discuss

### Nutrition heuristic: Cycle healthy options

17 июля, 2019 - 15:44
Published on July 17, 2019 12:44 PM UTC

It also doesn't help that different people have contradictory theories, e.g. meat, eggs, and diary are either very important to eat, or very important to avoid. More precisely, the best form of meat is fish. Except you shouldn't eat fish, because they are full of deadly mercury. Villiam, Nutrition is Satisficing

I really liked this comment, so I wanted to hone in on it.

Suppose we have 3 options for a protein source for dinner: Chicken, steak, or fish. They're all considered to be pretty good sources of protein, by most doctors, but they all have downsides:

• Steak is linked to heart disease,
• fish contains higher levels of heavy metals, and
• chicken is just kind of bland.

What should you do?

A nutrition plan is a little bit like a stock portfolio, in that you can diversify away risk by investing a small amount in several different companies at once. However, many of the risks we talk about with nutrition are linked to the overconsumption of specific foods. That means that diversification is super effective!

So the smart satisficer's move is to have all three, but cycle them. If we're planning for the week, we might do chicken Monday-Wednesday-Friday, and then on the other 4 days alternate between steak and fish. This does require a bit of planning on your part, but if you stick to variations between just a handful of pretty-good sources, I think it's pretty feasible.

(Note that I'm considering "Grilled chicken breast is really healthy, but I had it every night for three weeks and I'm never touching it again" as a strong long term and short term risk of overconsumption. In fact, since this seems to happen often with the healthiest foods we eat, I would strongly advise you to update on this prior as well: Extremely healthy foods need to be cycled in and out with their only very healthy, but tasty, options.)

Discuss

### Intertheoretic utility comparison: examples

17 июля, 2019 - 15:39
Published on July 17, 2019 12:39 PM UTC

A previous post introduced the theory of intertheoretic utility comparison. This post will give examples of how to do that comparison, by normalising individual utility functions.

The methods

All methods presented here obey the axioms of Relevant data, Continuity, Individual normalisation, and Symmetry. Later, we'll see which ones follow Utility reflection, Cloning indifference, Weak irrelevance, and Strong irrelevance.

Max, min, mean

The maximum of a utility function u is maxs∈Su(s), while the minimum is mins∈Su(s). The mean of u ∑s∈Su(s)/||S||.

• The max-min normalisation of [u] is the u∈[u] such that the maximum of u is 1 and the minimum is 0.

• The max-mean normalisation of [u] is the u∈[u] such that the maximum of u is 1 and the mean is 0.

The max-mean normalisation has an interesting feature: it's precisely the amount of utility that an agent completely ignorant of its own utility, would pay to discover that utility (as a otherwise the agent would employ a random, 'mean', strategy).

For completeness, there is also:

• The mean-min normalisation of [u] is the u∈[u] such that the mean of u is 1 and the minimum is 0.

The last two methods find ways of controlling the spread of possible utilities. For any utility u, define the mean difference: ∑s,s′∈S|u(s)−u(s′)|. And define the variance: ∑s∈S(u(s)−μ)2, where μ is the mean defined previously.

• The mean difference normalisation of [u] is the u∈[u] such that u has a mean difference of 1.

• The variance normalisation of [u] is the u∈[u] such that u has a variance of 1.

Properties

The different normalisation methods obey the following axioms:

Property Max-min Max-mean Mean-min Mean difference Variance Utility reflection YES NO NO YES YES Cloning indifference YES NO NO NO NO Weak Irrelevance YES YES YES NO YES Strong Irrelevance YES YES YES NO NO

As can be seen, max-min normalisation, despite its crudeness, is the only one that obeys all the properties. If we have a measure on S, then ignoring the cloning axiom becomes more reasonable. Strong irrelevance can in fact be seen as an anti-variance; it's because of its second order aspect that it fails this.

Discuss

### RAISE AI Safety prerequisites map entirely in one post

17 июля, 2019 - 11:57
Published on July 17, 2019 8:57 AM UTC

All RAISE projects have been discontinued, and a postmortem is in our plans. One of those projects was the AI Safety prerequisites online course originally announced here. We're sharing the curriculum here in almost plaintext format so that people can easily find and access it. There is also a GDoc here. (Note: two other products of RAISE are series of lessons on IRL and IDA. They don't have such neatly formed curriculums, and they are still accessible as online lessons.)

It was supposed to be a guide for helping people who want to get into AI safety research. It contains only foundations of math topics (Logic and proof, ZF(C) Set theory, Computability theory to be precise), which are more useful for agent foundations stuff and not useful for machine learning stuff. It was planned to be extended to cover more topics, but that never happened.

How to use this

The main path contains 20 levels. It is the recommended path through the curriculum. Its levels are actually short sequences of levels from the three other paths.

To see what textbooks are required to study a path, see its beginning. Computability theory and set theory paths require two paid textbooks.

13 out of 20 levels of the main path are covered by our online course, which is free (but still requires paid textbooks). To use the online course, register here. You might prefer to use it instead of the text below because it provides more sense of progress, contains solutions to the exercises, has some writing mistakes fixed, maybe feels like a less tedious thing, and provides some additional exercises which we don't think are important.

Credits

The curriculum was made by Erik Istre and Trent Fowler. People who created the online course are: Philip Blagoveschensky, Davide Zagami, Toon Alfrink, Hoagy Cunningham, Danilo Naiff, Lewis Hammond. Also these people contributed: Jacob Spence, Roland Pihlakas. Also, thanks to Grasple for providing their services for free.

Main path Level 1. Basic logic

The big ideas:

1. Sentential Logic
2. Truth Tables
3. Predicate Logic
4. Methods of Mathematical Proof

To move to the next level you need to be able to:

1. Translate informal arguments into formal logic.
2. Evaluate an argument as either valid or invalid.
3. Explain how to prove an implication/conditional, a conjunction, a disjunction, and a negation and know what this looks like informally (i.e. in words and not symbols).

Why this is important:

• This builds the basic knowledge you need to be able to produce and understand mathematical proof. A firm foundation in how logical machinery operates is the best way to be assured that a proof you produce or read is correct. This also teaches the basic methods by which a proof is produced.

Skill Guides for this Level:

Level 2. Basic set theory

The big ideas:

1. Axioms of Set Theory
2. Set Operations

To move to the next level you need to be able to:

1. Explain what a set is.
2. Calculate the intersection, union and difference of sets.
3. Prove two sets are equal.
4. Apply basic axioms of Zermelo-Fraenkel set theory.

Why this is important:

• Set theory has become entrenched as the basic language with which all mathematics can be discussed. While there are more estranged parts of set theory that will likely be irrelevant to you, a fluency in the basic materials of set theory is necessary to understand more advanced mathematics.

Skill Guides for this level:

Level 3. Set Theoretic Relations and Enumerability

The big ideas:

1. Ordered Pairs
2. Relations
3. Functions
4. Enumerability
5. Diagonalization

To move to the next level you need to be able to:

1. Define functions in terms of relations, relations in terms of ordered pairs, and ordered pairs in terms of sets.
2. Define what a one-to-one (or injective) and onto (or surjective) function is. A function that is both is called a one-to-one correspondence (or bijective).
3. Prove a function is one-to-one and/or onto.
4. Explain the difference between an enumerable and a non-enumerable set.

Why this is important:

• Establishing that a function is one-to-one and/or onto will be important in a myriad of circumstances, including proofs that two sets are of the same size, and is needed in establishing (most) isomorphisms.
• Equivalence relations and partial orderings are essential mathematical concepts which are powerful tools that can be used to analyze other mathematical objects or build new ones.
• Enumerability and non-enumerability introduces the difference between something being computable and non-computable.

Skill Guides for this level:

Level 4. Formal Semantics

The big ideas:

1. Formal Semantics
2. Model

To move to the next level you need to be able to:

1. Evaluate the truth value of logical sentences in a given model.
2. Build models for a set of logical sentences and then use those models to deduce information about the sentences.

Why this is important:

• An AI built on a formal system will reason based on some sort of proof and model theory. The former gives its methods of proof (which you’ll learn in the next chapter), and the latter its semantics.
• This level will give you your first sense of what models of logical sentences look like.

Skill guides for this level:

Level 5. Formal Proof

The big ideas:

1. Natural Deduction Proof System

To move to the next level you need to be able to:

1. Explain the difference between a formal system of proof and our informal notion of proof.
2. Derive proofs of logical formula in the system of natural deduction.

Why this is important:

• Learning mathematical proof is hard, as you may have experienced in levels 2 and 3. By learning a formal system of proof you will make your own thoughts more rigorous, understand the smallest details that need to be covered to perform a proof, and build your intuition for informal proof.

Skill guides for this level:

Milestone - You’re now a !

• You now understand the basics of logic and how they apply to proof.
• You can take an argument and peel away the content to look purely at its structural details.
• You understand the basics of building a model of formal sentences of logic.
• You know the basic building blocks of set theory.
• You now know there exist uncomputable functions, and you know how to rigorously define the concept of a function.

Take a moment to reflect on what you’ve learned so far. Remind yourself of the important concepts that you’ve come across, and mark down any concepts which are still giving you trouble. Determine whether you need to seek out further resources to clear up any confusions.

Level 6. Turing Machines and the Halting Problem

The big ideas:

1. Turing Machine
2. The Halting Problem

To move to the next level you need to be able to:

1. Describe a turing machine and write basic turing algorithms.
2. Give the basic idea of the halting problem.
3. Give the basic idea of the proof that the halting function isn’t computable.

Why this is important:

• This is the beginning of learning formal computability. This allows us to think about computation without the limits of a physical domain. The first step to reasoning about Artificial General Intelligence will be understanding how it can be done before we worry about its practical feasibility.
• The halting problem is a pervasive and annoying problem. It’ll come up again and again, in many disguises. If we could solve the halting problem, things would be easy. Since we can’t, things are hard. Understanding the limitations it imposes is important.

Skill guides for this level:

Level 7. Equivalence Relations and Orderings

The big ideas:

1. Equivalence Relations
2. Partitions
3. Orderings

To move to the next level you need to be able to:

1. Explain the relationship between equivalence relations and partitions.
2. Name two different kinds of orderings and the conditions on the ordering relations required for these kinds.

Why this is important:

• This level is about further building up your mathematical toolbox. While equivalence relations and orderings currently seem like random abstract notions, they are both very important. The former is gives you a way to instantiate different notions of equality, which is important in modeling. The latter appears in constructing models for another turing machine equivalent notion of computability, the lambda calculus.

Skill guides for this level:

Level 8. Abacus Computability and Mathematical Proof by Induction

The big ideas:

1. Proof by Induction
2. Abacus Computability

To move to the next level you need to be able to:

1. Explain mathematical proof by induction.
2. Explain the abacus machine and the differences between a turing machine and an abacus machine.
3. Build basic algorithms for an abacus machine.

Why this is important:

• Mathematical proof by induction is a very powerful technique. While you’ll first learn to use it in the context of natural numbers, you’ll soon see it has applications beyond this domain.
• An abacus machine gives you a higher level means of expressing algorithms, allowing you to abstract away from the minute details of strokes on a tape.

Skill guides for this level:

Level 9. The Natural Numbers in Set Theory and More Induction

The big ideas:

1. Representing Natural Numbers with Sets

To move to the next level you need to be able to:

1. Explain the relationship between inductive sets and the set theoretic construction of the natural numbers.
2. Use the method of mathematical induction to prove claims about the natural numbers.

Why this is important:

• This builds an understanding of how set theory is used to provide rigorous constructions of mathematical objects we usually take for granted, like the natural numbers. Learning how to provide these constructions is an important part of building models in model theory.

Skill guides for this level:

Level 10. Recursive Functions

The big ideas:

1. Primitive Recursive Functions
2. Recursive Functions
3. Primitive Recursive Sets
4. Recursive Sets

To move to the next level you need to be able to:

1. Translate the formal syntax of the recursive functions into an informal representation of a function and vice versa.
2. Explain the difference between primitive recursion and recursion.
3. Explain the difference between semi-recursive and recursive.

Why this is important:

• By this point in your FAI career, you may have developed some sort of fascination of the power of this idea of “recursion”. Well, now you’re learning that notion. It is powerful, but it can also get a bit complicated so take your time and on this concept.

Skill guides for this level:

Level 11. Set Theoretic Recursion

The big ideas:

1. The Recursion Theorem
2. Peano Axioms

To move to the next level you need to be able to:

1. State the recursion theorem and explain how this theorem coincides with our informal notion of recursion.
2. Use the recursion theorem to create new functions.
3. Explain what the axioms of Peano arithmetic are.

Why this is important:

• This is recursion...again! (In true recursive spirit.) However, it’s a bit different this time. This time we’re showing a way to look at recursion from a set theory perspective. We can take the recursive functions of computability as given and do some things with them, or we can use our set theoretic foundations to build the theory of recursion itself. Think of this as further justification and an assurance of what we’re doing.
• You’ll also get a look at what are known as “Peano’s axioms”. If we assume these axioms, or alternatively provide a set theoretic structure in which they hold, the claim is that these axioms can give us all our theorems of informal arithmetic in a formal system. Formalization of arithmetic will be important later for deriving limitative results about formal systems.

Skill guides for this level:

Level 12. The Equivalence of Different Notions of Computability

The big ideas:

1. Equivalence of Turing Machines and Recursive Computability

To move to the next level you need to be able to:

1. Describe the general process of coding turing machines.
2. Explain what a universal turing machine is.

Why this is important:

• Establishing the equivalence of independently developed methods of computation is a strong argument in favor of the Church-Turing thesis.
• The existence of a universal turing machine is a good thing. It made the general purpose computer that we’re so used to today a feasible invention. We don’t need 1000 specialized machines, but only one that can imitate those 1000.

Skill guides for this level:

Milestone - You’re now a !

• You now understand the basic abstract representations of computation.
• You know some of what we can expect from computers and also what we can’t expect from them.
• You know a lot more set theoretic tools like equivalence relations and orderings.
• You’ve seen the construction of the natural numbers from the perspective of set theory.
• You know about mathematical induction and you’ve used it!
• You know about recursion and you’ve used it!
• More abstractly, your mathematical maturity has greatly increased from when you began.

As last time, take a moment to reflect on what you’ve learned so far. Remind yourself of the important concepts that you’ve come across, and mark down any concepts which are still giving you trouble. Determine whether you need to seek out further resources to clear up any confusions.

Level 13. Isomorphisms

The big ideas:

1. Isomorphism

To move to the next level you need to be able to:

1. Define isomorphism between structures.
2. Prove that two structures are isomorphic.

Why this is important:

• Isomorphism is one of the most fundamental mathematical concepts. Finding isomorphisms allows you to move insights from one problem domain into another, which can be incredibly useful.
• However, it should be noted that “isomorphism” only rigorously applies when talking about algebraic structures, i.e. structures with relations and operations and a domain.

Skill guides for this level:

Level 14. Logic Review and The Relationship Between Computation and Logic

The big ideas:

1. First Order Logic Syntax
2. First Order Logic Semantics
3. The Relationship Between Logic and Computability

To move to the next level you need to be able to:

1. Read and translate the syntax for formal logic, and translate informal sentences to logic.
2. Build models to evaluate logical claims.
3. Explain how logic and turing machines/primitive recursion are related and how this leads to showing the undecidability of first-order logic.

Why this is important:

• This chapter reviews concepts from Levels 4 and 5 so it’s important for the same reasons covered there.
• The undecidability of logic has important implications for how an Artificial General Intelligence is limited (in much the same way we are!) in deriving proofs.

Skill guides for this level:

Level 15. Finite and Countable Sets

The big ideas:

1. Finite Set
2. Countable Set

To move to the next level you need to be able to:

1. State the Cantor-Bernstein Theorem.
2. Understand basic properties of finite and countable sets.
3. Define what a countable set is.
4. Prove that a set is of countable size.

Why this is important:

• The Cantor-Bernstein theorem may not have strict relevance for you. You can consider it part of your general mathematical well-being. It’s some set theory you really should know.
• Finite and countable sizes are what we as finite beings have “constructive” access to. We reason about other cardinalities, but it all becomes a little less graspable and tangible. These sizes of sets are important and probably give us everything we need in the real world. (For more on this: Computable Real Numbers)

Skill guides for this level:

Level 16 (elective). Linear Orders and Completing the Real Numbers

The big ideas:

1. Constructing the Real Numbers

To move to the next level you need to be able to:

1. Define when two linear orders are similar.
2. Explain how the concept of completeness motivates the real numbers.
3. Explain how Dedekind cuts generate completeness for the rational numbers.

Why this is important:

• If we look at the rationals from the perspective of completeness, we motivate the development and construction of the real numbers. This provides another useful example of a mathematical construction.
• In the current context, the real numbers are a prime example of an uncountable set.

Skill guides for this level:

Level 17. Basic Model Theory

The big ideas:

1. Models
2. Soundness
3. Completeness
4. Sequent Calculus

To move to the next level you need to be able to:

1. Define a model.
2. Explain the Lowenheim-Skolem theorem and the compactness theorem.
3. Use a Gentzen style system or sequent calculus to derive proofs.
4. Explain the concepts of soundness and completeness.

Why this is important:

• Model theory allows us to map a formal proof theoretic system into a defined domain where it hopefully applies. Whether or not it applies appropriately depends on the soundness and completeness results. This is all about asking “Do my formal deductions match up with something out there and do all of the somethings out there match up with my formal deductions?” As you may already know or will see in a few levels, the latter question is impossible to prove in formalized situations, but still worth pondering.

Skill guides for this level:

Milestone - You’re now a !

• You know the rigorous definition of an isomorphism.
• You now can explain the relationship between computability and logic.
• You understand way more about the size of things and way more sophisticated ways to count than you ever thought necessary.
• You can give a general outline of how to build the real numbers from the rationals.
• You can use a Gentzen style system to develop proofs.
• You can use soundness and completeness to demonstrate those Gentzen style proofs are “nice”.
• You know how to think about models more in-depth and about some of the more important theorems that apply to them.

As last time, take a moment to reflect on what you’ve learned so far. Remind yourself of the important concepts that you’ve come across, and mark down any concepts which are still giving you trouble. Determine whether you need to seek out further resources to clear up any confusions.

Level 18 (elective). A quick look at cardinal and ordinal numbers

To move to the next level you need to be able to:

1. Explain what a cardinal number is.
2. Demonstrate the general idea behind the proof of Cantor’s Theorem.
3. Explain what an ordinal number is.
4. Use the generalized notions of transfinite induction and transfinite recursion to prove statements about large sets.

Why this is important:

• This level is mainly for expanding what set theoretic tools you’re familiar with. While it is unlikely that cardinal theory or ordinal theory will directly play into AGI research, it still may crop up in unexpected places like model theory.
• Ordinal theory also sets you up to later learn what is known as “proof theoretic strength” of a formal system which is expressed in ordinal numbers and relies on transfinite induction.

Skill guides for this level:

Level 19. Arithmetization and Representation of Recursive Functions

The big ideas:

1. The Axiom of Choice
2. Arithmetization of Syntax
3. Mathematical Induction
4. Representability of Functions

To move to the next level you need to be able to:

1. Elective: State the axiom of choice and at least loosely explain it to someone with basic set theory.
2. Describe the process of generating Godel numbers for logical statements.
3. Define what it means for a recursive function to be representable in a system of arithmetic.

Why this is important:

• Being aware of the axiom of choice makes it easier to be aware when you are invoking it. This generally points out how the apparently simplest of assumptions can turn out to be just that: assumptions. We must be prepared to discover all of our assumptions working into our informal proofs before we think about applying them to a potentially hazardous problem like an AGI.
• The rest of this level is setting up the material for the oft-quoted Godel incompleteness theorems which place hard limits on what formal systems are capable of.

Skill guides for this level:

Level 20. Godel’s Incompleteness Theorems and Axiomatic ZFC

The big ideas:

1. Godel’s Incompleteness Theorems
2. Independence Result
3. Zermelo-Fraenkel Set Theory

To move to the next level you need to be able to:

1. Explain the diagonal lemma and why it leads to the limitative results of formal systems. Further, explain under what precise conditions we must be for these results to accurately apply.
2. Explain Godel’s first and second incompleteness theorems and the general procedure of their proofs.
3. Define what an independence result is and a particular example of such a result for ZFC.
4. Explain why Godel’s Incompleteness theorems also apply to a formal set theory like ZFC.

Why this is important:

• This level is all about formal systems and what they can’t do. These are the problems that will need to be circumvented, or at least shown to be irrelevant to a development of a formal AGI.

Skill guides for this level:

Milestone - You’re now a !

• You’re now a well-rounded set theorist.
• You can explain how to code up logical statements and give them Godel Numbers.
• You can explain why this leads to limitative results in formal systems and what these results mean.

As last time, take a moment to reflect on what you’ve learned so far. Remind yourself of the important concepts that you’ve come across, and mark down any concepts which are still giving you trouble. Determine whether you need to seek out further resources to clear up any confusions.

Logic and proof path

The main textbook used in this path is a free textbook "Forall x" version 1.4. download v1.4 another link for v1.4 latext version Note that numbering of exercises sometimes changes between versions.

Another resource used in this path is Introduction to Mathematical Arguments by Michael Hutchings.

Level 1. Basic logic
Level 2. Quantified logic. Introduction to mathematical arguments

This level on Grasple

• Read Chapter 4 of forall x, “Quantified Logic”
• Less Important Sections: “Definite Descriptions”
• Part A
• From Part B:
• Work at least 5, one with the word “all”, one with the word “some”, and one with the word “no”.
• From Part D:
• 1
• 5
• 6
• Part H
• Part I
• Part K
• All odd numbers.
• Read Introduction to Mathematical Arguments until the end of section 3
• Print/save/copy the table on page 9 and always have it handy
• Section 3.3 talks about groups, which you may have never heard of. Pay attention to the structure of the proof, since that is what we’re interested in here, and ignore the content if it seems confusing. Proof by uniqueness proceeds by saying “if I want to prove there is only one something, I assume I have two somethings and show that they turn out to be the same thing”.
• Work exercises:
• 1
• 2
• 3
• Extra Credit: 4
Level 3. Formal semantics basics

This level on Grasple

• Read Chapter 5 of forall x, “Formal Semantics”
• Work Exercises:
• Part B
• Part D
• Part F: 1, 2, 3, 5, 8
• Part G: 3, 6, 10
• Part H: 1, 4, 5, 9
• Part I: 1, 2, 3
• Extra Credit: Part J
Level 4. Formal proofs

This level on Grasple

• Read Chapter 6 of forall x, “Proofs”
• Part A
• Part B
• Part E: 1, 2, 5
• Part G: 1, 3, 4
• Part I
• Part J
• Part Q
• Part T: 1, 2, 4
• Part U: 1, 2, 3, 7, 8, 10
• Extra Credit: Part M
Level 5. Proof by induction

This level on Grasple

• Read Section 4 of Introduction to Mathematical Arguments , “Proof by Induction”
• This is a very powerful and important proof technique to get comfortable with. Make sure you understand the structure of this argument.
• Work the following exercises:
• 1
• 2
• 4
• Extra Credit:
• 4
• This will provide a review of the basic concepts of sets if you want a refresher/reinforcement.
• All of the exercises for the appendix are good checks on your knowledge of set theory.
• For 8, draw pictures to guess at new relationships and then prove them or disprove them by finding a counterexample.
Set theory path Level 1. Basic set theory

This level on Grasple

• In Introduction to Set Theory by Hrbacek and Jech
• Read Chapter 1 Section 1 “Introduction to Sets”
• Read Chapter 1 Section 2 “Properties”
• Read Chapter 1 Section 3 “The Axioms”
• Work the following exercises:

• 3.1
• 3.2
• This question is equivalent to asking you to prove:
• “If the Weak Axiom of Existence and the Comprehension Schema hold, then the empty set exists.”
• This is a general note to always try to translate a mathematical assertion into a straightforward logical statement. Once you have it in that logical statement, then you can rely on your logical knowledge to know general steps you need to do to prove that statement.
• 3.4

• Proving this requires a few steps that are not immediately obvious.
• First, prove there is a set that contains A and B. (What axioms implies this?)
• Second, prove there is a set that contains the elements of A and B. (Again, what axiom?)
• Finally, use the Comprehension Schema on this set to get the desired set.
• 3.5a

• This one also requires a few steps as with 3.4. To find these steps, work backwards.
• What instance of the comprehension schema do you need to make this set exist?
• What other set’s existence does that instance of the comprehension schema rely on?
• How do you make that one exist?
• Does this new set rely on another one for existence?
• Continue to move backwards until you can be sure that you have a set that exists. You will have to rely on the fact that the existence of A, B, and C is assumed.
• Extra Credit: 3.3

• Read Chapter 1 Section 4
• Work the following exercises:
• 4.1
• Do all formulas involving union, intersection, and difference.
• If you’re proving set equality between two arbitrary sets A and B, you are always starting with the assumption that you have an arbitrary element x in A and trying to show that it gets into B.
• Reread the axiom of extensionality and explain why you are trying to prove that “if x is in A then x is in B” and “if x is in B then x is in A”.
• Extra credit: The symmetric difference formulas.
• 4.2a
• How do you prove a chain of “if and only if”? Investigate what it means to prove for arbitrary sentences A, B, and C: A iff B iff C.
• 4.2b
• 4.3
• Extra Credit: 4.4
Level 2. Set theoretic relations

This level on Grasple

This level uses Introduction to Set Theory by Hrbacek and Jech.

• The level of abstraction will start getting a bit higher as we progress. We’ll be defining objects to define other objects to define other objects. Check yourself and make sure you know what each term refers to and what each terms means as the abstraction progresses.
• Read Chapter 2 Section 1 “Ordered Pairs”
• Work the following exercises:
• 1.1
• Part of 1.2
• It’s enough to work out the proof of existence for (a, b). This is to review your axioms.
• 1.3
• Extra Credit: 1.6
• This is mainly interesting to state the equivalent theorem. This illustrates that set theory can represent the same concept in multiple ways. The proof of this equivalent theorem is tedious and unenlightening after going through Theorem 1.2.
• Read Chapter 2 Section 2 “Relations”
• Work the following exercises:
• 2.3 Parts: a, b, c, d
• 2.4 Parts: a, d
• 2.6
• Extra Credit: 2.1
• Further axiom review.
• Read Chapter 2 Section 3 “Functions”
• Pay special attention to:
• Definition 3.1
• Definition 3.3
• Definition 3.7
• A function that is one-to-one is also called injective, and a function that is onto is also called surjective. A function that is both is called a one-to-one correspondence or a bijection.
• Work the following exercises:
• 3.1
• 3.2
• 3.3
• 3.4a
• 3.5
• 3.6
• Extra credit:
• 3.10
• 3.11
Level 3. Equivalence relations and orderings

This level on Grasple

Resources: Introduction to Set Theory by Hrbacek and Jech

• Read Chapter 2 Section 4 “Equivalences and Partitions”
• Pay special attention to:
• Definition 4.1
• Definition 4.3
• Definition 4.6
• Theorem 4.10
• Work the following exercises:
• Prove Theorem 4.10.
• 4.1
• Not necessary to produce proofs for this one, play with the relations and see what you come up with.
• 4.2
• Read Chapter 2 Section 5 “Orderings”
• Pay special attention to:
• Definition 5.2
• Definition 5.5
• Work the following exercises:
• 5.1
• 5.3
• 5.5
• 5.7
• 5.12
• Extra Credit:
• 5.13
• 5.14
Level 4. Natural numbers and induction

This level on grasple

Resources: Introduction to Set Theory by Hrbacek and Jech

• Read Chapter 3 “Natural Numbers” Section 1 “Introduction to Natural Numbers”

• Work the following exercises:
• 1.1
• For the second half of 1.1, assume there is such a z and reason from there. Refer to Introduction to Mathematical Arguments Proof by Contradiction and/or Uniqueness proofs.
• Read Chapter 3 Section 2 “Properties of Natural Numbers”

• Pay special attention to:
• The Induction Principle
• Work the following exercises:
• 2.1
• 2.2
• 2.3
• 2.4
• 2.6
• 2.7
• Extra Credit:
• 2.11
• 2.12
• 2.13
• Read Chapter 3 Section 3 “The Recursion Theorem”

• Pay special attention to:
• The Recursion Theorem
• Work the following exercises:
• 3.1
• 3.2
• Extra Credit:
• 3.5
• 3.6
• Read Chapter 3 Section 4 “Arithmetic of Natural Numbers”

• Pay special attention to:
• The Peano Axioms
• Work the following exercises:
• 4.1
• 4.2
• 4.3
• 4.4
• 4.5
• Extra Credit:
• 4.7
• 4.8
Level 5. Set theoretic recursion

This level at grasple

Resources: Introduction to Set Theory by Hrbacek and Jech

• Read Chapter 3 Section 3 “The Recursion Theorem”
• Pay special attention to:
• The Recursion Theorem
• Work the following exercises:
• 3.1
• 3.2
• Extra Credit:
• 3.5
• 3.6
• Read Chapter 3 Section 4 “Arithmetic of Natural Numbers”
• Pay special attention to:
• The Peano Axioms
• Work the following exercises:
• 4.1
• 4.2
• 4.3
• 4.4
• 4.5
• Extra Credit:
• 4.7
• 4.8
Level 6. Operations, structures, isomorphism

This level on Grasple

Resources: Introduction to Set Theory by Hrbacek and Jech

• Read Chapter 3 Section 5 “Operations and Structures”
• Pay special attention to:
• Definition 5.6
• This is a more abstract notion of isomorphism. It is dependent not only on a bijection, but making sure that structural features are preserved (like relations or functions).
• Work the following exercises:
• 5.1
• 5.4
• 5.6
• 5.12
Level 7. Cardinality. Finite and countable sets

Resources: Introduction to Set Theory by Hrbacek and Jech

• Read Chapter 4 “Finite, Countable, and Uncountable Sets” Section 1 “Cardinality of Sets”
• Pay special attention to:
• Theorem 1.6 (Cantor-Bernstein)
• Work the following exercises:
• 1.1
• 1.2
• 1.5
• Extra Credit:
• 1.10
• 1.11
• 1.12
• Read Chapter 4 Section 2 “Finite Sets”
• Pay special attention to:
• Lemma 2.2
• This lemma is also known as the pigeonhole principle in combinatorics. In fact, most of this chapter is teaching you the basic counting methods of finite combinatorics, which is important in probabilities of events with finitely many outcomes.
• Work the following exercises:
• 2.1
• 2.2
• 2.3
• 2.5
• Read Chapter 4 Section 3 “Countable Sets”
• Pay special attention to:
• Corollary 3.6
• This gives you a vague idea of how “small” countable sets are compared to uncountable.
• Work the following exercises:
• 3.1
• 3.2
• 3.3
• Extra Credit:
• 3.5
• 3.6
Elective: Level 8. Linear orderings. Completeness. Uncountable sets

Resources: Introduction to Set Theory by Hrbacek and Jech

• Read Chapter 4 Section 4 “Linear Orderings”
• Work the following exercises:
• 4.1
• 4.3
• Read Chapter 4 Section 5 “Complete Linear Orderings”
• Pay special attention to:
• Theorem 1.3
• Definition 5.5
• Definition 5.6
• Work the following exercises:
• 5.1
• For a little review of proof by contradiction.
• 5.2
• Extra credit:
• 5.5
• 5.8
• Read Chapter 4 Section 6 “Uncountable Sets”
• Pay special attention to:
• Cantor’s Proof of Theorem 6.1
• Work the following exercises:
• 6.1
Elective: Level 9. Cardinal numbers

Resources: Introduction to Set Theory by Hrbacek and Jech

• Don’t get stuck in this level. Become familiar with these ideas and then move on. If you’re working in computation, countable sets are mostly what you worry about. It’s nice to be aware of these properties of cardinality to be more set theoretically well-rounded.
• Read Chapter 5 “Cardinal Numbers” Section 1 “Cardinal Arithmetic”
• Pay special attention to:
• Theorem 1.8 (Cantor’s Theorem)
• Theorem 1.9
• Don’t spend too much time on:
• The operations of cardinal arithmetic. Unlikely to be very important to you.
• Work the following exercises:
• 1.5
• 1.7
• Read Chapter 5 “Cardinal Numbers” Section 2 “The Cardinality of the Continuum”
Elective: Level 10. Ordinal numbers. Axiom of replacement. Transfinite induction and recursion

Resources: Introduction to Set Theory by Hrbacek and Jech

• Same as the previous level. Ordinal numbers won’t be terribly important except in some model theoretic cases. Don’t get bogged down, this is probably way more ordinal theory than you’ll need. Become familiar with the ideas, and then move on.
• Read Chapter 6 “Ordinal Numbers” Section 1 “Well-Ordered Sets”
• Work the following exercises:
• 1.4
• Read Chapter 6 Section 2 “Ordinal Numbers”
• Pay special attention to:
• Definition 2.2
• Work the following exercises:
• 2.1
• 2.2
• 2.3
• Read Chapter 6 Section 3 “The Axiom of Replacement”
• Work the following exercises:
• 3.1
• Read Chapter 6 Section 4 “Transfinite Induction and Recursion”
• Pay special attention to:
• 4.1 Transfinite Induction
• 4.5 Transfinite Recursion
• Neither section 5 or section 6 is likely to be useful to you. Read if you’re interested but don’t feel compelled to.
Elective: Level 11. Axiom of choice

Resources: Introduction to Set Theory by Hrbacek and Jech

• This level is for further mathematical enrichment. Anyone claiming to be skilled in set theory should be aware of the axiom of choice and it’s consequences. However, as with the last two levels, you only need familiarity. Don’t get stuck here. The next level will return to set theory that is more important to you.
• Read Chapter 8 “The Axiom of Choice” Section 1 “The Axiom of Choice and Its Equivalences”
• Optional Read Chapter 8 Section 2 “The Use of the Axiom of Choice in Mathematics”
Level 12. ZF(C) set theory

Resources: Introduction to Set Theory by Hrbacek and Jech

• We are again back to something that matters to AGI research: formal systems.
• Read Chapter 15 “The Axiomatic Set Theory” Section 1 “The Zermelo-Fraenkel Set Theory With Choice”
• Read Chapter 15 Section 2 “Consistency and Independence”
• The existence of independence results is part of the heart of the problem in building an AGI. How do we make a formal system capable of self-extension so as to cover as many problem domains as possible when we know our formal systems always have blind spots?
• Read Chapter 15 Section 3 “The Universe of Set Theory”
• This last section should give you an idea of what work is still being done in set theory, and that the field is far from closed. More generally, it shows that formal systems are in constant need of being tuned and pushed to their limits if we are to keep improving them.
Computability theory path

This path uses Computability and Logic (Fifth Edition) by Boolos, Burgess and Jeffrey. An instructor's manual is available, it has hints for many problems.

Level 1. Enumerability and diagonalization

This level on grasple

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• Pay special attention to:
• Example 1.2
• Example 1.3
• Example 1.9
• Example 1.13
• Work the following exercises:
• Note that to demonstrate a set is enumerable is to describe a function (or list) that will eventually list any element of the set. This description must be precise enough that anyone who inspects your enumeration becomes convinced that every element will in fact be listed.
• 1.1
• 1.2
• 1.3
• 1.5
• Hint: Example 1.9
• Extra credit: 1.7
• Diagonalization is a concept that will begin to pop up in a lot of places. In general, it sketches out the boundaries of what is logically possible. It’ll come up again, so start trying to wrap your head around it.
• Pay special attention to:
• Theorem 2.1 and how it is proved
• Work the following exercises:
• 2.1
• Hint: Refer to the proof of Theorem 2.1
• Another hint: Refer back to your solution of exercise 1.5
• Extra credit: 2.13
Level 2. Turing machines and the halting problem

This level on grasple

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• Read Chapter 3 “Turing Machines”
• Work the following exercises:
• 3.1
• 3.2
• 3.3
• Extra Credit:
• 3.5
• Read Chapter 4 “Uncomputability” Section 1 “The Halting Problem”
• Optional Read Chapter 4 Section 2 “The Productivity Function”
• Work the following exercises:
• 4.1
• 4.2
• Extra Credit:
• 4.5
Level 3. Abacus computability

This level on grasple

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• Read Chapter 5 “Abacus Computability”
• Work the following exercises:
• 5.1
• 5.2
• 5.4
• Extra Credit:
• 5.3
• 5.5
Level 4. Recursive functions

This level on grasple

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• Read Chapter 6 “Recursive Functions”
• While reading this chapter and before the exercises:
• What is the difference between a primitive recursive function and a recursive function?
• Practice unpacking the formal recursion syntax with functions given in the text. Do the reverse; practice tracing the steps to write them in formal syntax.
• Work the following exercises:
• 6.1
• 6.3
• Extra Credit:
• 6.2
• 6.7
• Read Chapter 7 “Recursive Sets and Relations” Sections 1 and 2
• Optional Read Chapter 7 Section 3 “Further Examples”
• Work the following exercises:
• 7.1
• 7.3
• 7.5
• Extra Credit
• 7.9
Level 5. The equivalence of different notions of computability

This level on grasple

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• Read Chapter 8 “Equivalent Definitions of Computability”
• This chapter’s main importance is in understanding the process of proving equivalence. Go through the proofs a couple time to ensure you are understanding. You don’t need to be able to reproduce them, but be able to follow.
• Extra Credit Exercise:
• 8.1
Level 6. First order logic

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• This level works out to be a review of first order logic and semantics. This also introduces you to a very important method of using proof by induction in logic.
• Read Chapter 9 “A Precis of First-Order Logic: Syntax”
• Work the following problems:
• 9.1
• 9.2
• 9.3
• 9.4
• Extra Credit:
• 9.6
• 9.7
• 9.8
• Read Chapter 10 “A Precis of First-Order Logic: Semantics”
• Work the following problems:
• 10.1
• 10.2
• 10.3
• 10.4
• 10.7
• Extra Credit:
• 10.8
• 10.12
• 10.14
Level 7. Undecidability of first order logic

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• Read Chapter 11 “The Undecidability of First-Order Logic”
• Work the following exercises:
• 11.1
• 11.2
• Extra Credit:
• 11.12
• 11.13
Level 8. Models, their existence. Proofs and completeness.

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• Work the following exercises:
• 12.1
• 12.2
• 12.8
• 12.15
• Extra Credit:
• 12.6
• 12.9
• 12.19
• Read Chapter 13 “The Existence of Models”
• Familiarity with the ideas in this chapter is encouraged, but it is nonessential to spend a significant amount of time here.
• Read Chapter 14 “Proofs and Completeness”
• Work the following exercises:
• 14.1
• 14.2
• 14.3
• 14.5
• Extra Credit: 14.9
Level 9. Arithmetization. Representability of recursive functions

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• This level builds the machinery to construct the proofs to Godel’s Incompleteness Theorems.
• Work the following exercises:
• 15.1
• 15.2
• 15.5
• Extra Credit:
• 15.9
• Read Chapter 16 “Representability of Recursive Functions”
• Section 16.4 is a must. It is worth knowing the comparison between the system presented in this book and what else you might see in the literature.
• Work the following exercises:
• 16.1
• 16.2
• 16.3
• 16.10
• 16.11
• Extra Credit:
• 16.8
• 16.9
• 16.21
Level 10. Indefinability, undecidability, incompleteness. The unprovability of consistency.

Resources: Computability and Logic Fifth Edition by Boolos, Burgess, Jeffrey

• The most important thing in the chapters of this level is to take the proofs slowly and understand them bit by bit.
• Read Chapter 17 “Indefinability, Undecidability, Incompleteness”
• Work the following exercises:
• 17.1
• Read Chapter 18 “The Unprovability of Consistency”

Discuss