# Новости LessWrong.com

A community blog devoted to refining the art of rationality
Обновлено: 44 минуты 17 секунд назад

### CPH meetup 10/10/19

10 октября, 2019 - 18:38
Published on October 10, 2019 10:08 AM UTC

This is intended as a space for people curious about the world and critical of the methods of understanding it. It is primarily aimed at people interested in applying Bayesian Rationality(LessWrong, CFAR, etc.) to their critical methods but also welcomes Slate Star Codex readers, and others curious about the world.

Last time the community seemed excited about a community member sharing their view on something they are knowledgeable about, and so without further ado I would like to present this weeks activity.

Activity: Community Presentation
Carl Dybdahl will be doing a presentation on the causes of gender dysphoria and transsexuality, where he will be talking about Blanchard's typology, research into gender issues, critiques of the model, and things that may be interesting to study in the future.

Schedule:
19.00 Informal meet and greet
19:30 Activity begins
21:00 Socialize

This is the second meetups in a series that will last until December.

We are meeting in the aquarium.

Discuss

### What is the real "danger zone" for food?

10 октября, 2019 - 14:00
Published on October 10, 2019 11:00 AM UTC

In my post about keeping dry food warm in school lunches I wrote that "The general rule is that hot food shouldn't be below 140F (60C) for more than two hours, because substantial bacteria can grow, and the closer it is to 100F (37C)the worse it is." Someone said they had heard 130F (55C) was the limit, and asked where this rule came from, and I read more about it.

Bacteria that's most dangerous to us generally thrive best around body temperature, so the farther you get from 100F (37C) the better. Food safety material generally describes this as a "danger zone", between refrigerator temperature and cooking temperature. This is what's represented in the FDA's Food Code:

3-501.19.B.1 Time as a Public Health Control: Time - maximum up to 4 hours:

The FOOD shall have an initial temperature of 5C (41F) or less when removed from cold holding temperature control, or 57C (135F) or greater when removed from hot holding temperature control

It gives more detail in section 3-501.16's "The Safety of the Time as a Public Health Control Provision from Cooking Temperatures (135F or above) to Ambient":

FDA conducted in-house laboratory experiments to test the safety of the existing TPHC provisions of 4 hours without temperature control starting with an initial temperature of 135F or above. Clostridium perfringens was chosen to represent a worst case scenario pathogen for foods allowed to cool from cooking temperatures to ambient without temperature control, because its spores can survive normal cooking procedures, it can grow at relatively high temperatures (>120F) and it has a short lag period. C. perfringens spores were inoculated into foods that were cooked and then cooled to yield a cooling curve that would promote outgrowth as quickly as possible. The growth data suggest that the existing 4-hour TPHC provision will be safe for 6 hours after cooking, with the additional 2-hour margin of safety built-in for consumer handling.

There's a great chart in The "Danger Zone" Reevaluated by Frank Bryan, showing how long food can spend at various temperatures:

Since the time I care about is 4hr from packing to eating, even under the food code guidelines it should be safe if it starts at cooking or refrigerator temperatures. To be safe, though, I'm going to keep using the thermal mass thermos approach.

Discuss

### Testing the Efficacy of Disagreement Resolution Techniques (and a Proposal for Testing Double Crux)

10 октября, 2019 - 10:18
Published on October 10, 2019 7:18 AM UTC

Introduction

I will describe a procedure for testing the efficacy of disagreement resolution techniques (DRTs) together with methodologies for inducing their use (induction methods). DRTs are structured games (in the loosest senses of the terms) that involve conversation, and are aimed at helping participants who disagree about some topic either figure out the truth about that topic, or come to better understand each others' positions and the sources of their disagreement. An induction method is just any way of trying to get someone to use a DRT.

I am writing up a description of this procedure here because I plan to use it to test and find DRTs, and I would like to get feedback before I go out and do that. I originally came up with this procedure in order to test Double Crux, and I still plan to. I will describe the first step of that plan in the second half of this post. The first half of the post explains the general procedure and some frames I think might be useful for understanding the second half of the post.

I would also like to invite others to use the general procedure for whatever seems like a good idea to them. It seems fairly obvious to me now, but something something hindsight.

I would like any kind of feedback that you think might make the methodology I describe herein better. I am particularly interested in feedback on the specific procedure I plan to use for testing Double Crux, but feedback on the general procedure would also be great. Any feedback or advice on the statistical analysis of the results of my test procedure for Double Crux would also be appreciated.

General Procedure

Step 0:

Gather participants and filter them for desired characteristics. Ignorance of the DRT to be tested should be verified for all participants regardless of the aims of those conducting the study.

Inform accepted participants of the nature of the study: that they will be scored according to a proper scoring rule; that they may have to take a module; discuss a controversial question with a stranger, etc.

Step 1:

Have participants assign credence distributions over the possible answers to multiple choice questions. Such questions should have definite correct answers, and it should be hard or impossible for participants to look up the answers.

Step 2:

Pair participants according to disagreement measured by total variation distance.

Step 3:

Randomly assign participants to one of three groups.

Control Group 1: Members are given no special instructions.

Control Group 2: Member are given basic advise on how to have useful conversations with people who disagree with you.

Treatment Group: Members will be induced to use the DRT to be tested during the conversation in step 4 using the induction method to be tested.

Step 4:

Inform members of all three groups who they were paired with and what question the pair disagreed about. Members of the first two groups are instructed to have a conversation with their partner with the aim of becoming more confident about what the right answer is.

For the third group, the induction method is applied either before or during their conversations depending on its design. Members of the third group are instructed to use the DRT being tested in order to figure out the right answer to the question they were assigned.

Have all three groups conduct their conversations.

Step 5

Have participants assign a new credence distribution over the possible answers to the multiple choice questions they discussed in step 4.

Step 6:

Pay participants according to a proper scoring rule scored on the credence distributions they assigned in step 5.

Measurement

The second kind of data I would like to collect is the degree to which a pair's beliefs converged after the conversation regardless of the truth of the answer on which they converged. I will measure this by taking the total variation distance of their credence distributions before their conversation, and subtracting from that number the total variation distance between the credence distributions they assigned after their conversation.

Constraints on Multiple Choice Questions

The multiple choice questions used should be questions for which participants are unlikely to already know the correct answer. They should also be questions for which it is reasonable to expect that people might make incremental progress on figuring out the correct answer through conversation, eg: "will president Trump be reelected in 2020" would be fine, but "what is Joe Smith's sister's name" would not.

The questions should also have the property that if you find out the right answer, it is not completely trivial to convince others of the right answer. Counterintuitive physics puzzles satisfy this criteria, but most raven's progressive matrices questions do not.

It might be good for there to be a set of neutral questions, such as counterintuitive physics or logic puzzles, or questions about how long it took for spider silk to evolve, as well as a set of controversial or politically sensitive questions, such as who will be elected or nominated for some important political office, or whether the GDP of a country went up or down after a particular policy was implemented.

DRTs are not Individuated by their Induction Methods

The original example of a DRT that I had in mind was Double Crux. Having anchored on Double Crux, the alternative possible DRT-induction method pairs I imagined I might later test shared Double Crux's canonical pedagogical structure: you are taught the technique by someone who knows it or maybe read about it, you practice it some, and then you use it with other people. The space of possible DRT-induction method pairs is much larger than this would suggest.

DRTs are not always transmitted to someone before the first time they are used. Some methods of induction allow the first time someone uses a DRT to be simultaneous with the first time they learn how it works or beforehand.

One example of an induction method that allows participants to use a DRT before they learn how the DRT works is having a trained facilitator facilitate their conversation according to the norms of the DRT.

Another might be using a program that guides participants through a discussion. Maybe it keeps track of their Cruxes; keeps track of the various topic branches and sub conversations; the different arguments that have been made and what their premises are; keeps track of users' credences and asks them to update them explicitly when a new argument is inputted, etc. If a DRT were codified in such a program, participants might not have to be preemptively instructed in how to use the DRT. The program might be sufficiently intuitive and give participants enough instruction on its own. This would have the added bonus that they could later use that program without needing a trained facilitator around.

I could imagine designing a program like this for Double Crux, but also for other DRTs that are not Double Crux. Of course, you could also design alternative DRTs and transmit them in the way that Double Crux is normally transmitted. This means that whether an accompanying induction method is applied before or during the first time participants use a DRT does not depend much on the nature of the DRT itself.

The general procedure does not test the efficacy of DRTs in isolation; it tests them together with a particular method of induction. If a DRT totally fails to get any interesting result when we try to induce it with one method, that does not mean that the DRT itself is inefficacious. It might be that the induction method used failed to induce the DRT's use, or that it induced its use but failed to transmit the most important fragments of the DRT.

Specific Plans to Test Double Crux Induced by an Online Training ModuleProcedure

Step 0:

I will collect participants on positly.com. I will filter them for not already knowing what Double Crux is. I may also filter them for having completed a bachelor's degree depending on how difficult the final set of questions and the concepts involved in the training module turn out to be. Postily participants are already filtered by Positly for being decent study participants in general.

Step 1:

I will have them take a questionnaire that contains several multiple choice questions. Some of these will be predictions about politically sensitive topics. Some of these will be physics or logic puzzles. Some will be numerical estimation questions like "how many miles of road are there in Africa". I will ask them to assign credence distributions over the multiple choices for each question.

I will then ask them if they would like to participate in a larger study. I will first explain what sorts of things they can expect to do during that study and what sorts of compensation they can expect to earn, if they say yes, I will collect their emails and record their data on a spreadsheet.

Step 2:

I will pair participants according to disagreement, preferring larger disagreements to smaller ones. Again, this will be measured by total variation of distance.

Step 3:

I will then randomly assign pairs to one of three groups:

Control Group 1: Members of this group will not be given any specific advice.

Control Group 2: Members of this group will be given general advice on how to have useful conversations with people they disagree with. I will tell them that they should try to see each other as cooperating to solve a puzzle, that they should see themselves as partners on the same team, that they should try really hard to understand why the other person had such different beliefs from them, and that they should try to ask each other a lot of questions.

Treatment Group: Members of this group will be asked to complete an online training module that is intended to teach them some critical fragment of the Double Crux technique, and asked to Double Crux with their partners in step 4. (More on the design of this module later.)

Step 4:

Participants will be asked to communicate with the other member of their pair via a video chat service, or a live chat service. (More on this later.) They will be asked to speak to each other for as long as they would like, but for no less than 30 minutes.

Step 5:

I will then have participants assign a new credence distribution over the multiple choice answers for the question they were assigned to discuss in step 4.

Step 6:

I may offer participants extra compensation to record or send a log of their conversation.

I will then recruit people who can credibly claim to assess Double Crux performance, and have them rate logs or recordings of the conversations for Double Cruxiness. They will rate these logs on a scale from 1 to 10. They should give a 1 to any conversation they consider to not be an instance of Double Crux at all, a 5 to a conversation that seems average Double Cruxy for conversations between people who disagree and know about Double Crux, and a 10 to any conversation that seems like a totally paradigmatic instance of Double Crux.

If I decide to do this, I will try to design the module without the use of the word "crux" so as to minimize the chance that participants use the term in their logs, as that would certainly tip off the raters.

Step 7:

Reward participants according to a Briar score scored on the distributions they assigned in step 5.

Measurements

I plan to measure the same quantities mentioned above. I will measure the amount of information gained by each participant from the conversation, and I will measure the degree to which each pair converged.

Text Chat or Video Chat?

I am genuinely unsure about what medium I should ask people to conduct their conversations through. Video chat is closer to in person conversation, and I personally find the extra bandwidth useful, especially when I am trying to have a cooperative conversation. However, it would be easier to have participants send logs that can be rated and analyzed if the conversations are mediated through text. Text also offers the advantage of providing a visible log of the conversation for the participants to look over, which is basically the same as giving participants more working memory.

I would be interested to hear other peoples' thoughts on this.

Why have People Rate the Double Cruxiness of the Conversations?

I would like to take logs of the conversations and have people rate them for Double Cruxiness so that if I get a negative result, it is more informative. Without step 6, if it turned out that participants in the treatment group got about as much information out of their conversations as members of either control group, we might interpret this as being primarily evidence that the module used in the experiment does not teach Double Crux.

But if we know that people who understand Double Crux rate the participants in the treatment group as using Double Crux more than those in the control groups, then we can rule out that explanation.

If the raters rate the treatment group's Double Cruxiness above the level of the control groups, then that suggests that the module teaches Double Crux. As such, a negative result would be some evidence against the efficacy of the Double Crux technique itself.

If the raters rate the treatment group's Double Cruxiness near the level of the control groups, this means that the module does not teach Double Crux, or at least it does not teach it any better than giving some basic advice about how to have useful disagreements.

In that case, the treatment group scoring about as well as control group 2 would not be much evidence against the efficacy of Double Crux itself, although it would certainly be further evidence that the module does not successfully induce the technique.

A positive result would suggest that those people I consider relative authorities on Double Crux are not reliable raters of the degree to which the technique is being used, or are not sensitive to the important aspects of its use. I think this would be an interesting thing to learn and further investigate. It might help with the design of future DRTs or training modules.

I might ultimately leave this step out of a pilot study. I would be more likely to include step 6 in a later follow up to the pilot.

If I did decide to include step 6, I would make sure that nobody, including my self, knew how well the treatment group did compared to the control groups until after the rating was done.

Designing the Module

Although I use Double Crux often, I do not think of myself as a qualified instructor of the technique. I would like the module to be designed primarily by people who can credibly claim to have successfully taught Double Crux multiple times. Preferably it would be designed by people who have also put a lot of thought into figuring out what parts of the technique are most important, and what makes it work. I will try to recruit or contract some such folks to help me design the module.

If they turn out to be too expensive for my current budget, then I will just try my best at designing the module, plagiarizing from the people I would have hired as much as possible, and asking for as much feedback as I can handle along the way.

I expect that the module should include examples, diagrams, and questions that check for understanding. I would prefer for it to be no longer than 30 minutes, but it might have to be.

Regardless of who ends up designing it, I will be asking for feedback on the module from other people.

Pilot Study

My current plan is to pay people 20 usd just for participating, since it is a rather intensive study. (It might take up to an hour and a half to complete. I will start out offering a lower price and see what happens, but I would be happy to go as high as 20 usd.) I will also offer up to 10 usd in monetary rewards for updating towards the right answer.

I may or may not offer an additional reward for sending a log if I decide to include step 6 in the pilot study at all. This would be more likely if I ended up asking participants to use video chat instead of text, since recording video is harder than copying and pasting.

I would like to have at least 30 participants in each group for the pilot study.

Statistical Analysis

My plan is to use both standard significance analysis, specifically a Welch test, and also to use the Bayesian method, BEST, described here. In this section, I will use a standard difference of means test since it is easier to explain, but the Welch test gives similar results. You can verify their similarity yourself using this online calculator.

There are three distributions we are interested in. The distributions of our measurements for the control group 1 population, the control group 2 population, and the treatment group population. The analysis is the same for either, except that the sample for the convergence measure will have half as many data points. I will focus on the evidence measure here, since it is what I am more interested in, especially for the pilot study. I will call their respective means &#x3BC;1, &#x3BC;2, and &#x3BC;t.

There are two hypotheses we want to test, and two corresponding null hypotheses.

H0:&#x3BC;1−&#x3BC;t≤0

H0:&#x3BC;1−&#x3BC;t>0

And also:

H0:&#x3BC;2−&#x3BC;t≤0

H0:&#x3BC;2−&#x3BC;t>0

It might also be interesting to compare &#x3BC;1 and &#x3BC;2.

The significance analysis for all such tests is the same, so I will arbitrarily pick the first one. The means of two distributions being equal is equivalent to the means of their sample distributions being equal: &#x3BC;¯x1−&#x3BC;¯xt=0 . The distributions of &#x3BC;¯x1 and &#x3BC;¯xt are approximately normally distributed by the central limit theorem, and so their difference is also normally distributed. To estimate the standard deviation of the difference distribution we use the pooled variance formula:

&#x3C3;¯x1−¯xt=√s21n1+s2tnt

Estimating the sample standard deviations to be about 4 bits of information (higher than I expect them to be) each, with a sample size of 30 in each group, that gives us the following critical values for each significance level:

For &#x3B1;=.2 we should reject the null hypothesis at approximately ¯x1−¯x2=0.87 .

For &#x3B1;=.15 we should reject at approximately ¯x1−¯x2=1.07 .

For &#x3B1;=.1 we should reject at approximately ¯x1−¯x2=1.33 .

If instead we guess sample standard deviations of 2 bits, then we get:

For &#x3B1;=.2 we should reject the null hypothesis at approximately ¯x1−¯x2=0.43 .

For &#x3B1;=.15 we should reject the null hypothesis at approximately ¯x1−¯x2=0.66 .

For &#x3B1;=.05 we should reject the null hypothesis at approximately ¯x1−¯x2=0.85.

I am optimistic that I can get better a sample mean difference greater than 0.43, but that would not be significant at the liberal &#x3B1;=.2 level unless the standard deviation is 2 or less. I think it is possible the pilot study will get a sample difference of greater than 0.87. I am less optimistic about greater than 1.07, and I think better than 1.33 is unlikely. Of course, the sample standard deviations will almost certainly be higher or lower than my current best guesses.

My expectations about what results I am likely to get are pure speculation, but they are my best guesses, and I do take my guesses about the sample standard deviations to be genuinely conservative. I will likely later end up laughing at how optimistic or pessimistic I previously was.

It might be worth pointing out that with a sample size of about 115 in each group, and sample standard deviations around 3, the critical value for &#x3B1;=0.5 is about 0.7, which I would be relatively optimistic about beating.

As state above, I also plan to use BEST to do a Bayesian analysis of the results. I will give a few example priors and their corresponding posteriors, and hopefully also provide a program in R or Python that allows one to input a prior and get a posterior as output.

If there is a better way to do this than the Welch test that does not assume equal variances of the relevant population distributions, or a better bayesian approach, I am all ears. I would also be interested in hearing about alternatives to the BEST approach .

I might be missing something important. This is just the bog standard approach I found after googling and asking around for a bit.

In any case, I plan to make all data (aside from the conversation logs) publicly available (while preserving anonymity of course). I would encourage folks to analyze that data however they like and let me know what they find.

What Happens After you get your Result?

If I get a positive result, I will look for further funding to do a larger study. If I get a positive result on a stronger study, I will have a module that has been empirically shown to helps pairs of people who disagree get more information out of their conversations. I plan to distribute any such module for free. I will of course also try to look for other DRT-induction methods with larger effect sizes.

If I get a negative result on the pilot study, I will still make the data publicly available, as well as the module, and let people make their own judgments on how that result should update them. I may release a program that allows one to input a prior over a parameter space and gives the appropriate posterior as output.

I will then continue to systematically test DRTs and methods for inducing their use.

This is all part of my grander plot to rigorously test, develop, and disseminate, systematic methods for improving the art of human rationality. I am not going to give up on that because of one negative result.

With that, I would like to thank Daniel Filan, Vaniver (Matthew Graves), Oliver Habryka, Luke Raspkoff, Katja Grace, and Eliezer Yudkowsky, for their feedback and/or their much appreciated encouragement.

Discuss

### Thoughts on "Human-Compatible"

10 октября, 2019 - 08:24
Published on October 10, 2019 5:24 AM UTC

The purpose of this book is to explain why [superintelligence] might be the last event in human history and how to make sure that it is not... The book is intended for a general audience but will, I hope, be of value in convincing specialists in artificial intelligence to rethink their fundamental assumptions.

Yesterday, I eagerly opened my copy of Stuart Russell's Human Compatible (mirroring his Center for Human-Compatible AI, where I've worked the past two summers). I've been curious about Russell's research agenda, and also how Russell argued the case so convincingly as to garner the following acclamations from two Turing Award winners:

Human Compatible made me a convert to Russell's concerns with our ability to control our upcoming creation—super-intelligent machines. Unlike outside alarmists and futurists, Russell is a leading authority on AI. His new book will educate the public about AI more than any book I can think of, and is a delightful and uplifting read.—Judea Pearl

This beautifully written book addresses a fundamental challenge for humanity: increasingly intelligent machines that do what we ask but not what we really intend. Essential reading if you care about our future. —Yoshua Bengio

Bengio even recently lent a reasoned voice to a debate on instrumental convergence!

Bringing the AI community up-to-speed

I think the book will greatly help AI professionals understand key arguments, avoid classic missteps, and appreciate the serious challenge humanity faces. Russell straightforwardly debunks common objections, writing with both candor and charm.

I must admit, it's great to see such a prominent debunking; I still remember, early in my concern about alignment, hearing one professional respond the entire idea of being concerned about AGI with a lazy ad hominem dismissal. Like, hello? This is our future we're talking about!

But Russell realizes that most people don't intentionally argue in bad faith; he structures his arguments with the understanding and charity required to ease the difficulty of changing one's mind. (Although I wish he'd be a little less sassy with LeCun, understandable as his frustration may be)

More important than having fish, however, is knowing how to fish; Russell helps train the right mental motions in his readers:

With a bit of practice, you can learn to identify ways in which the achievement of more or less any fixed objective can result in arbitrarily bad outcomes. [Russell goes on to describe specific examples and strategies] (p139)

He somehow explains the difference between the Platonic assumptions of RL and the reality of a human-level reasoner, while also introducing wireheading. He covers the utility-reward gap, explaining that our understanding of real-world agency is so crude that we can't even coherently talk about the "purpose" of eg AlphaGo. He explains instrumental subgoals. These bits are so, so good.

Now for the main course, for those already familiar with the basic arguments:

The agenda

Please realize that I'm replying to my understanding of Russell's agenda as communicated in a nontechnical book for the general public; I also don't have a mental model of Russell personally. Still, I'm working with what I've got.

Here's my summary: reward uncertainty through some extension of a CIRL-like setup, accounting for human irrationality through our scientific knowledge, doing aggregate preference utilitarianism for all of the humans on the planet, discounting people by how well their beliefs map to reality, perhaps downweighting motivations such as envy (to mitigate the problem of everyone wanting positional goods). One challenge is towards what preference-shaping situations the robot should guide us (maybe we need meta-preference learning?). Russell also has a vision of many agents, each working to reasonably pursue the wishes of their owners (while being considerate of others).

I'm going to simplify the situation and just express my concerns about the case of one irrational human, one robot.

There's fully updated deference:

One possible scheme in AI alignment is to give the AI a state of moral uncertainty implying that we know more than the AI does about its own utility function, as the AI's meta-utility function defines its ideal target. Then we could tell the AI, "You should let us shut you down because we know something about your ideal target that you don't, and we estimate that we can optimize your ideal target better without you." The obstacle to this scheme is that belief states of this type also tend to imply that an even better option for the AI would be to learn its ideal target by observing us. Then, having 'fully updated', the AI would have no further reason to 'defer' to us, and could proceed to directly optimize its ideal target.

which Russell partially addresses by advocating ensuring realizability, and avoiding feature misspecification by (somehow) allowing for dynamic addition of previously unknown features (see also Incorrigibility in the CIRL Framework). But supposing we don't have this kind of model misspecification, I don't see how the "AI simply fully computes the human's policy, updates, and then no longer lets us correct it" issue is addressed. If you're really confident that computing the human policy lets you just extract the true preferences under the realizability assumptions, maybe this is fine? I suspect Russell has more to say here that didn't make it onto the printed page.

There's also the issue of getting a good enough human mistake model, and figuring out people's beliefs, all while attempting to learn their preferences (see the value learning sequence).

Now, it would be pretty silly to reply to an outlined research agenda with "but specific problems X, Y, and Z!", because the whole point of further research is to solve problems. However, my concerns are more structural. Certain AI designs lend themselves to more robustness against things going wrong (in specification, training, or simply having fewer assumptions). It seems to me that the uncertainty-based approach is quite demanding on getting component after component "right enough".

Let me give you an example of something which is intuitively "more robust" to me: approval-directed agency.

Consider a human Hugh, and an agent Arthur who uses the following procedure to choose each action:

Estimate the expected rating Hugh would give each action if he considered it at length. Take the action with the highest expected rating.

Here, the approval-policy does what a predictor says to do at each time step, which is different from maximizing a signal. Its shape feels different to me; the policy isn't shaped to maximize some reward signal (and pursue instrumental subgoals). Errors in prediction almost certainly don't produce a policy adversarial to human interests.

How does this compare with the uncertainty approach? Let's consider one thing it seems we need to get right:

Where in the world is the human?

How will the agent robustly locate the human whose preferences it's learning, and why do we need to worry about this?

Well, a novice might worry "what if the AI doesn't properly cleave reality at its joints, relying on a bad representation of the world?". But, having good predictive accuracy is instrumentally useful for maximizing the reward signal, so we can expect that its implicit representation of the world continually improves (i.e., it comes to find a nice efficient encoding). We don't have to worry about this - the AI is incentivized to get this right.

However, if the AI is meant to deduce and further the preferences of that single human, it has to find that human. But, before the AI is operational, how do we point to our concept of "this person" in a yet-unformed model whose encoding probably doesn't cleave reality along those same lines? Even if we fix the structure of the AI's model so we can point to that human, it might then have instrumental incentives to modify the model so it can make better predictions.

Why does it matter so much that we point exactly to be human? Well, then we're extrapolating the "preferences" of something that is not the person (or a person?) - the predicted human policy in this case seems highly sensitive to the details of the person or entity being pointed to. This seems like it could easily end in tragedy, and (strong belief, weakly held) doesn't seem like the kind of problem that has a clean solution. this sort of thing seems to happen quite often for proposals which hinge on things-in-ontologies.

Human action models, mistake models, etc. are also difficult in this way, and we have to get them right. I'm not necessarily worried about the difficulties themselves, but that the framework seems so sensitive to them.

Conclusion

This book is most definitely an important read for both the general public and AI specialists, presenting a thought-provoking agenda with worthwhile insights (even if I don't see how it all ends up fitting together). To me, this seems like a key tool for outreach.

Just think: in how many worlds does alignment research benefit from the advocacy of one of the most distinguished AI researchers ever?

Discuss

### Examples of Categories

10 октября, 2019 - 05:19
Published on October 10, 2019 1:25 AM UTC

Before we go through any more conceptual talk, let's explore some examples of categories to get a better feel of what they are.

Categories do two things:

1) They function as mathematical structures that capture our intuition about how models of cause-and-effect (composition) work in general.

2) They generalize other mathematical structures that capture our intuition about how specific models work.

Obviously, 1 and 2 are related. We model particular pieces of reality with particular mathematical structures (e.g., we model shapes with geometry), so generalizing how we model the world and generalizing fields of mathematics are highly related tasks.

Because I'm assuming my audience isn't familiar with any mathematical structures, we can't just start tossing out examples. Essentially the art of "categorifying" a mathematical structure is as simple as identifying what are the objects and what are the associatively composable morphisms. See here for a list of examples (scroll down to the table), which ought to give you an idea that many different kinds of mathematics really are just examples of categories on some level.

To give just one example of a mathematical structure generalized by category theory, you might be relatively familiar with the category known as Set, the category whose objects are sets and whose morphisms are functions—functions being very nice maps between sets. Functions compose, as we've already seen, and you can check that this composition is associative, so, voila, we've "categorified" a mathematical structure.

Set as category:

Objects: sets

Morphisms: functions

Rule of composition: function composition, e.g., 2x2

Associativity: 2(x2)=(2×1)x2

Since Set is pretty important, we'll discuss it in a separate post to make sure we're all on the same page about sets and functions. But hopefully this gives you an idea of what it looks like to take a "math thing" and make it into a category.

***

Even if you're not a math expert, you probably know that 2 is bigger than 1. That's all the level of expertise required to understand a very important type of a category called a poset, which is category-speak for "partially ordered set." A partially ordered set is a set with an order than is at least partial.

A poset is any category for which there is at most one morphism between any two objects. I.e., if you have A→B in a category, then you do not also have B→A in the same category, unless B is merely a relabeling of A. (In which case the morphism is the identity morphism, as a poset also rules that there can only be at most one morphism between any object and itself, and since every object must have an identity morphism, there isn't any other possibility.)

A poset is partially ordered in the sense that, since A→B but not B→A, you could think of A and B as being in an order. For example, you could represent natural numbers, e.g., 1,2,3, etc., as objects and have the morphism be "less than or equal to," i.e., the ≤ sign. Then A→B means "A≤B." Hence the category has an order—the objects are ordered by how big they are.

(Why less than or equal to and not simply less than? Because we need the identity morphism, and a number can't be less than itself. But it can be less than or equal to itself—by being equal to itself!)

Posets are partial orders in the sense that there can be at most one morphism between A and B. However, there does not need to be a morphism between them at all! For example, say your objects are sports players, and your morphism refers to "less than or equal to in skill." Then A→B would mean that that the basketball player A is less than or equal to in skill than basketball player B. However, suppose, that C is a soccer player. Then morphisms like A→C or B→C wouldn't really make sense. Hence, the orders in the category are only partial—they do not go through the entire category.

When you do have at least and at most one morphism between A and B, that's called a total order. The natural numbers are in fact totally ordered by size. One is less than or equal to two, two is less than or equal to three, etc. (Examples of total orders are probably easier to think of than partial orders, but, as we'll see later, partial orders are sufficient to do the things we really care about having orders for, without needing to go so far as to necessitate that the order be total. I.e., partial orders let us assume less to get the same basic outcome for what we'll be doing later.)

Posets are really important for a couple of reasons. The main one is that posets are the simplest interesting kind of category. Posets are interesting because they can have morphisms between different objects, and morphisms are where all the interesting stuff in a category lie, as we'll see in a couple of posts. However, because there is at most one morphism between any two objects in a poset, posets stay fairly tame and simple. If you're ever confused by a concept in category theory, it's pretty much always a good idea to apply it to a poset so that you can see the idea in its simplest interesting form.

We'll also see that posets are very useful for studying what it means to do things in the best possible way—which is what adjunction, arguably the key concept of category theory, is all about.

***

One of the nice things about categories is how easily visualized their underlying structure is. For example, take a poset like 1→2→3 and another poset like a→b→c. These are clearly very similar posets. (In fact, they're isomorphic.)

Despite the superficial differences, we clearly have "basically the same thing" going on with these two categories. So let's get abstract (which is what category theory is all about). Look at this category:

∙→∙→∙.

We've gotten rid of the labels on the objects and just replaced them with generic nodes. Using this framework, we can forget concerns about what specific category we're working with and just look at the general types of categories that can exist.

The most basic kind of category is the empty category, a category with no objects and therefore no morphisms. Obviously, we can't exactly depict this category visually—there's nothing to depict.

The "next" kind of category is a single object with just the identity morphism. Leaving the identity morphism implicit, it looks like this:

Growing more complex, we can add in a second object, but no morphisms between objects, just two objects and their identity morphisms.

∙∙

A category with just identity morphisms is called discrete—the objects aren't connected to each other, hence the term. (Note that all discrete categories are posets—there is clearly at most one morphism between any two objects.)

Turning the above diagram into a poset (in this case a total order) is as easy as adding an arrow.

∙→∙

Here's something called a "commutative square. I'll add in labels to make it easier to follow.

Af→Be↓↓hX→gY

The commutative square is a very important and famous shape in category theory. It commutes in the sense that the paths Ae→Xg→Y and Af→Bh→Y are equivalent. That is to say, if you start at A, you can go right then down to get to Y, or down then right to get to Y, and it does not matter which path you choose. This is exactly analogous to how the knight moves in chess: typically (always? or does it not hold at the edge of the board?) there are two paths the knight can follow to get to a particular square from its current position, and it does not matter which path you choose.

In fact, it matters so little that you can just jump straight to the square, you don't have to trace out the "L" that the knight does. Similarly, you can just jump straight from A to Y without actually going down one of the two paths: that's composition.

A lot of important claims in category theory can be boiled down to saying that a particular diagram, often a square, commutes. For example, the basic definition of a natural transformation has at its center a commutative square—and properly defining natural transformations was the original motivation for developing category theory! (We'll talk about natural transformations quite a bit later.)

From here we can branch out in lots of different ways in terms of adding morphisms and objects, but you should get the general idea. Whenever you can think of a model that can be boiled down to nodes and arrows, it's probably a category. You should know that categories like the two categories depicted here are totally valid. You can have an infinite number of morphisms between two objects going back and forth. In the end, as long as the morphisms compose (associatively), you can do pretty much anything you want. We'll get a clearer view of exactly how to talk about this once we've discussed sets and functions in the next post.

Discuss

### Long Term Future Fund application is closing this Friday (October 11th)

10 октября, 2019 - 03:44
Published on October 10, 2019 12:44 AM UTC

Just a reminder that the Long Term Future Fund application is closing in two days. In past rounds we received 80% of our applications the day of the deadline, but I am not sure whether that is contingent on us reminding everyone that the application is closing soon, so I will continue creating these reminder posts for now.

As a reminder, here is an excerpt from the application form of what kinds of grants we are looking for:

We are particularly interested in small teams and individuals that are trying to get projects off the ground, or that need less money than existing grant-making institutions are likely to give out (i.e. less than ~$100k, but more than$10k). Here are a few examples of project types that we're open to funding an individual or group for (note that this list is not exhaustive):

+ To spend a few months (perhaps during the summer) to research an open problem in AI alignment or AI strategy and produce a few blog posts or videos on their ideas
+ To spend a few months building a web app with the potential to solve an operations bottleneck at x-risk organisations
+ To spend a few months up-skilling in a field to prepare for future work (e.g. microeconomics, functional programming, etc).
+ To spend a year testing an idea that has the potential to be built into an org.
We are also interested in applications for larger projects, or potential future long-term organizations that require more than 100k of funding. You can find more details on the kind of project we are likely to fund on the fund page (in particular I recommend reading our past recommendation writeups): https://app.effectivealtruism.org/funds/far-future Discuss ### Minimization of prediction error as a foundation for human values in AI alignment 9 октября, 2019 - 21:23 Published on October 9, 2019 6:23 PM UTC I've mentioned in posts twice (and previously in several comments) that I'm excited about predictive coding, specifically the idea that the human brain either is or can be modeled as a hierarchical system of (negative feedback) control systems that try to minimize error in predicting their inputs with some strong (possibly un-updatable) prediction set points (priors). I'm excited because I believe this approach better describes a wide range of human behavior, including subjective mental experiences, than any other theory of how the mind works, it's compatible with many other theories of brain and mind, and it may give us an adequate way to ground human values precisely enough to be useful in AI alignment. A predictive coding theory of human values My general theory of how to ground human values in minimization of prediction error is simple and straightforward: • Neurons form hierarchical control systems. • Those control systems aim to minimize prediction error via negative feedback (homeostatic) loops. • The positive signal of the control system occurs when prediction error is minimized; the negative signal of the control system occurs when prediction error is maximized. • There is also a neutral signal when there is insufficient information to activate the positive or negative signal "circuitry". • cf. feeling/sensation is when the mind makes a determination about sense data, and sensations are positive, negative, or neutral • "Good", "bad", and "neutral" are then terms given to describe the experience of these positive, negative, and neutral control signals, respectively, as they move up the hierarchy. I've thought about this for a while so I have a fairly robust sense in my mind of how this works that allows me to verify it against a wide variety of situations, but I doubt I've conveyed that to you already. I think it will help if I give some examples of what this theory predicts happens in various situations that accounts for the behavior people observe and report in themselves and others. • Mixed emotions/feelings are the result of a literal mix of different control systems under the same hierarchy receiving positive and negative signals as a result of producing less or more prediction error. • Hard-to-predict people are perceived as creepy or, stated with less nuance, bad. • Familiar things feel good by definition: they are easy to predict. • Similarly, there's a feeling of loss (bad) when familiar things change. • Mental illnesses result from failures of neurons to set good/bad thresholds appropriately, to update set points at an appropriate rate to match current rather than old circumstances, and from sensory input issues causing either prediction error or internally correct predictions that are poorly correlated with reality (this broadly including issues related both to sight, sound, smell, taste, touch and to mental inputs from long term memory, short term memory, and otherwise from other neurons). • Desire and aversion are what it feels like to notice prediction error is high and for the brain to take actions it predicts will lower it either by something happening (seeing sensory input) or not happening (not seeing sensory input), respectively. • Good and bad feel like natural categories because they are, but ones that are the result of a brain interacting with the world rather than features of the externally observed world. • Etc. Further exploration of these kinds of cases will help in verifying the theory via whether or not adequate and straightforward applications of the theory can explain various phenomena (I view it as being in a similar epistemic state to evolutionary psychology, including the threat of misleading ourselves with just-so stories). It does to some extent hinge on questions I'm not situated to evaluate experimentally myself, especially whether or not the brain actually implements hierarchical control systems of the type described, but I'm willing to move forward because even if the brain is not literally made of hierarchical control systems the theory appears to model what the brain does well enough that whatever theory replaces it will also have to be compatible with many of its predictions. Hence I think we can use it as a provisional grounding even as we keep an eye out for ways in which it may turn out to be an abstraction that we will have to reconsider in the light of future evidence, and that work we do based off of it will be amendable to translation to whatever new, more fundamental grounding we may discover in the future. Relation to AI alignment So that's the theory. How does it relate to AI alignment? First note that this theory is naturally a foundation of axiology, or the study of values, and by extension a foundation for the study of ethics, to the extent that ethics is about reasoning about how agents, each with their own (possibly identical) values, interact. This is relevant for reasons I and more recently Stuart Armstrong have explored: Stuart has been exploring one approach by grounding human values in an improvement on the abstraction for human values used in inverse reinforcement learning that I think of as a behavioral economics theory of human values. My main objection to this approach is that it is behaviorist: it appears to me to be grounded in what can be observed from external human behavior by other agents and has to infer the internal states of agents across a large inferential gap, true values being a kind of hidden and encapsulated variable an agent learns about via observed behavior. To be fair this has proven an extremely useful approach over the past 100 years or so in a variety of fields, but it also suffers an epistemic problem in that it requires lots of inference to determine values, and I believe this makes it a poor choice given the magnitude of Goodharting effects we expect to be at risk from with superintelligence-levels of optimization. In comparison, I view a predictive-coding-like theory of human values as offering a much better method of grounding human preferences. It is • parsimonious: the behavioral economics approach to human values allows comparatively complicated value specifications and requires many modifications to make it reflect a wide variety of observed human behavior, whereas this theory lets them be specified in simple terms that become complex by recursive application of the same basic mechanism; • requires little inference: if it is totally right, only the inference of measuring neuron activity creates room for epistemic error within the model; • captures internal state: true values/internal state is assessed as directly as possible rather than inferred from behavior; • broad: works for both rational and non-rational agents without modification; • flexible: even if the control theory model is wrong, the general "Bayesian brain" approach is probably right enough for us to make useful progress over what is possible with a behaviorist approach such that we could translate work that assumes predictive coding to another, better model. Thus I am quite excited about the possibility that predictive coding approach may allow us to ground human values precisely enough to enable successfully aligning AI with human values. This is a first attempt to explain what has been my "big idea" for the last year or so now that it has finally come together enough in my head that I'm confident presenting it, so I very much welcome feedback, questions, and comments that may help us move towards a more complete evaluation and exploration of this idea. Discuss ### Expected Value- Millionaires Math 9 октября, 2019 - 17:50 Published on October 9, 2019 2:50 PM UTC I credit reading this post years ago as the first step towards making me instinctively THINK in terms of expected value when considering opportunities. I consider it to be among the best "Why" resources for thinking in expected value, and wish it was more known among the LW crowd. Discuss ### On Collusion - Vitalik Buterin 9 октября, 2019 - 17:45 Published on October 9, 2019 2:45 PM UTC ...if there is a situation with some fixed pool of resources and some currently established mechanism for distributing those resources, and it’s unavoidably possible for 51% of the participants can conspire to seize control of the resources, no matter what the current configuration is there is always some conspiracy that can emerge that would be profitable for the participants....This fact, the instability of majority games under cooperative game theory, is arguably highly underrated as a simplified general mathematical model of why there may well be no “end of history” in politics and no system that proves fully satisfactory; I personally believe it’s much more useful than the more famous Arrow’s theorem, for example. I've found this post quite useful in thinking about mechanism design and problems of designing stable systems. Discuss ### Regularly Scheduled: Day-of Reminders 9 октября, 2019 - 14:00 Published on October 9, 2019 11:00 AM UTC Two years ago I wrote Regularly Scheduled, a tool to make it easier for people to handle repeated events: I mostly use it for a recurring dinner we host with some friends from college. It works well, and has been running with basically no maintenance. After twice now forgetting I was hosting dinner in the few days between RSVPing "yes" and people showing up at my house, however, I realized it needed to be sending reminder emails day-of. I added these today, and they look like: They only go to people who RSVP'd yes; they're a reminder to come not a reminder to RSVP. If you have an event you'd like to set up with it, feel free! Let me know if you run into any issues and I'll try to get the code back into my head. Discuss ### Categories: models of models 9 октября, 2019 - 08:46 Published on October 9, 2019 2:45 AM UTC Let me clarify what I mean when I say that math consists of nouns and verbs. Think about elementary school mathematics like addition and subtraction. What you learn to do is take a bunch of nouns—1, 2, 3, etc.—and a bunch of verbs—addition, subtraction—and make sentences. “1 + 2 = 3.” When you make a sentence like that, what you're doing is taking an object, 1, and observing how it changes when it interacts—specifically, adds—with another object, 2. You observe that becomes a 3. Just like how you can observe a person (object) bump their head (interaction) into a wall (other object) and the corresponding change (equals sign): the bluish bump emerging on their forehead. Well, it turns out that no matter how far you go in math, you’re still taking objects and observing how they change when they interact with other objects. You learn about other kinds of numbers like 2.5 and 3.333 repeating, and other kinds of interactions like multiplication and division. Eventually things start getting more abstract. You learn about matrices and the kinds of interactions they can have. You learn about sets and functions in a deep way. Complex numbers, topologies, etc. But you never get away from nouns and verbs. What are we going to do with all of these nouns and verbs? Well, at some point, when you have a bunch of things that feel similar in an important way, you often want to take a step back, lump all of the individual things together, and talk about them in general. Why do we have the concept of fruit? Because we live in a world where apples, oranges, lemons, papayas, etc., exist. Why do we have the concept of numbers? Because we have a bunch of them, not just 1, 2, and 3, but also 2.5, 3.333 repeating, irrational numbers, and even complex numbers. Why do we have the concept of “operations?” Because we have addition, subtraction, multiplication, and division. Even though all of these give you very different answers for the same input of numbers, (1 + 2 does not equal 1 - 2 does not equal 1 times 2 does not equal 1 divided by 2), they still have something very similar in common, so it’s worth coming up with the concept of “operation” to study them independently of their individual characteristics. Just like how we can talk about “fruit” without having to reference the shape of an apple. If you lived in a world with only apples, you wouldn’t have the concept of fruit. If you lived in a world where the only kind of thing was a rock, you wouldn’t have the concept of nouns. If you lived in a world where the only kind of math-thing was numbers, category theorists wouldn’t have come up with the concept of objects. (They wouldn't have come up with category theory at all!) And what is the use of generalization? A bird’s-eye view of something lets you see things from new perspectives, potentially changing your concept of the thing entirely. When you think of numbers as various ways you can count stuff on your fingers, you can inch your way past the natural numbers to negatives, fractions and irrational numbers. But complex numbers will throw you for a total loop—they just don’t seem to be relatable to “amounts of things” in a way that you’re ever likely to see in the world. In fact, complex numbers made no sense to me until I learned what numbers actually are, at which point everything clicked into place. The generalization helped me understand the specific case—complex numbers are just another type of number, like how apples are just another type of fruit. Generalization is helpful when life stops throwing convenient examples at you. For example, say you want to grow new kinds of fruit that have never existed. Having a concept of fruit is necessary to conceiving of that idea. Life's not going to give you examples of fruit that have never existed! You have to explore the conceptual space of all fruit. (In fact, this experience with complex numbers, years ago, probably inspired these posts. I got the idea that if you just bothered to take a really long time to explain the mathematics, the average person could probably grasp quantum physics. This series is the category-theory version of that original idea.) Category theory exists for the same reason the concept of fruit does: there are lots of individual things that have certain commonalities that make generalization an appealing idea. Apples and oranges are very different on the surface and very similar deep down. Natural numbers and complex numbers are very different on the surface and very similar deep down. Category theory goes one step wider. It emerges when people look at entire fields of math like “algebra” and “topology” and notices that, while they’re very different on the surface, they seem to be very similar deep down. Many other fields of mathematics seemed to also share these deep similarities, and so gradually they all became mere examples of the generalization, that generalization being a category. (A category being something we'll define by the end of this post.) Just like how apples and oranges become mere examples of the generalization that is “fruit.” And one of those commonalities is that all of these superficially disparate fields of mathematics study things and interactions between those things. I.e., they study objects and morphisms. That might sound really general. And it is! And yet, just like with the general definition of fields that lets us understand complex numbers, we can learn really interesting things from this super-general perspective. (Specifically, the Yoneda lemma and adjunction.) But right now you are having to take my word for the idea that many different fields of math can be thought of as studying “nouns and verbs.” So let’s look at things from a different perspective. Even if you don’t know higher maths, you probably know things like, “pouring milk in my cereal will make my cereal soggy.” So “milk + cereal = soggy cereal.” Seems awfully...mathematical. Why does math exist, anyway? Well, there’s lots of ways to answer that question, so let’s rephrase it: why do mathematicians exist? Or even better, why do mathematicians get paid? It certainly isn’t for the joy of doing math. Instead, “mathematician” is a job that you can actually get paid to do because math is very useful for modeling our reality. So why does math boil down to objects and morphisms so often? Probably for the same reason English boils down to nouns and verbs: we use language to discuss reality, and reality seems to boil down to nouns and verbs. Take birds, for example. They are birds, so they’re nouns (objects). And they do stuff like fly, tweet, lay eggs, eat, etc. I.e., verbs (morphisms). Whatever you may or may not know of maths, you definitely know a thing or two about reality. You’ve been living in it your whole life. The rest of this post will take common sense ideas about creating models of our reality, break them down into their most abstract components, and end up with some simple rules that any model should follow. We’ll discover that those components and rules are exactly the components and rules that define a category. *** You know a lot of models, even if most of them would never make it into a textbook. You use these models to navigate reality. For example, a sentence like "Alice pushes Bob" is a model of reality. By themselves, those letters in that order are meaningless. But because you can use them to make predictions (specifically, you're inferring the speaker's state of mind), the sentence is correspondingly a model of (that particular part of) reality. Sentences themselves exist, so we can model them too. You can think of a sentence structure as a way of modeling a sentence. As for models themselves, a model is a way of abstracting away from some details of whatever you're studying so that other features become more salient. For example, you can model a specific cat, Mr. Peanuts, as just a cat, neglecting specific features like his name, his color, etc., so that other, more general features of cats, like the tendency to meow, eat tuna fish, and utterly despise humans become more salient. That is to say, what will Mr. Peanuts do in [specific situation]? Hard to say unless you know him personally. But if we ask, "What will a cat do in [specific situation]?" you might have a decent guess. The cat's not going to start barking, for one. You have a model of cats inside your head, and this lets you make predictions, at the expense of specificity. Sentence structure models specific sentences like "Alice pushes Bob" in terms of their abstract components: "Noun verbs noun." You might be surprised to learn that you can make predictions with sentence structure. For example, one of the rules of grammar is that a prepositional phrase ends with a noun. So let's say I ask you, "Who will the soccer player kick the ball to?" You don't know the answer—I haven't even given you a list of people to choose from. Let's also say that you don't know what the word "who" means, so you don't even know what I'm asking. In fact, let's say you don't know what any of the words in the sentence means, only their parts of speech. You certainly aren't going to give the specific answer I'm looking for. But you do know sentence structure! You know that "to" is a preposition, so the phrase it opens up must end with a noun. So who is the soccer player going to kick the ball to? Answer: a person, place, or thing—a noun. This is not a brilliant answer. But it is an answer—a correct one, in fact! This super abstract model of sentences let you figure things out even when you didn't know what any of the specific words meant. So sentences are models, and sentence structure is a model of a model, and models are really useful: they let you predict things. But sentence structure is a model of just one kind of model—sentences themselves. What if we wanted a model of all kinds of different models—sentences, sentence structure, scientific models, models from sensory data, mathematical models, everything? What if we wanted to model modeling itself? Why, we'd turn to category theory. So what are the general qualities of all models? First of all, every model has things—objects—and actions that the objects to do other objects—morphisms. For example, you can model a person by seeing them. That is to say, you hit them with some photons and form a picture of them based on the result. Every model is ultimately found in how one object changes another object—hence the term "morphism." (Notice how sentences, which themselves boil down to nouns and verbs, can be thought of as models of anything. After all, it's hard to imagine any kind of phenomenon that can't be expressed in a sentence, even if you have to make up some new words to do it. Effectively, the reason we use category theory to form models of models of everything instead just sticking with English is because we can define our category as obeying certain rules or axioms that correspond to how models ought to work, while English can say pretty much anything. The rest of this post discusses these rules.) So categories—models of models—consist of objects and morphisms. But what are the rules we require our objects and morphisms to obey so that they match up with our ideas of how models of reality ought to work? One is that every object ought to be, in some sense, a perfect model of itself. Think about what models do: they transform how we view objects. If I tell you someone is a murderer, this changes how you view them because you apply a different model. Loosely, you could say that you apply the "murderer" morphism to the person in question. Well, every object ought to have a model that you can apply that doesn't transform the object at all—not because you aren't transforming, but because there's no effect, like multiplying by 1. For example, if you apply the "Mr. Peanuts" morphism to Mr. Peanuts, this shouldn't change anything: Mr. Peanuts is already Mr. Peanuts. This morphism is, in a certain clear sense, the identity of Mr. Peanuts. So perhaps unsurprisingly, in category theory, we say that every object has an identity morphism that "does nothing." For an arbitrary object .mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} A, its identity morphism is written 1A. That's because it's like multiplying by 1—you are multiplying, it's just that nothing changes. (But just like multiplying both sides by 1 helps you solve equations, the identity morphism is extremely useful.) What does it actually mean, mathematically, for the identity morphism to "do nothing?" To answer that, let's look at another requirement we'd expect of a general model of models: compositionality. Say Alice pushes Bob, and Bob bumps into Charles. Let's diagram this as Af→Bg→C. Hopefully the interpretation of this diagram is obvious. We could say that Alice affects Bob, and Bob affects Charles—that is to say, Bob's state is a model of Alice's action on Bob, and Charles's state is a model of Bob's action on Charles. (I know we used p to stand for push last time, but f and g are the typical "generic morphism" symbols in category theory, and it's better to get used to using generic symbols than trying to name them after things all the time. Here, f is Alice pushing Bob, and g is Bob bumping into Charles.) But we also might have an intuition that Alice affects Charles. Sure, she does so indirectly through Bob, but clearly Charles's state is a model of Alice's actions—actions which are in some sense actions on Charles. That is to say, Alice pushed Charles just as clearly as she pushed Bob. Effect flows through. We're all just models of the original state and laws of the universe, whatever they were. So whenever we have something like Af→Bg→C, we should really expect to also have something like Ah→C. Moreover, this morphism h should be equal to what happened when Alice pushed Bob and then Bob bumped into Charles. After all, that is what Alice did to Charles. Let's use the symbol ∘ to mean "following." So if we write g∘f , that reads as "g following f." (Yes, that means you read it right to left, in effect. Get used to it, every field has at least one counterintuitive piece of notation.) We would then say that h=g∘f. More to the point, the idea is that whenever you have something that looks like Af→Bg→C, regardless of what these objects and morphisms are meant to stand in for, then you necessarily have Ah→C such that h=g∘f. This is what composition looks like as a diagram. (The backwards E means "there exists," and the dotted line, redundantly, means the same thing.) An example of ∘ that you are hopefully familiar with is function composition. (We'll cover it in another post anyway.) Say you have 2x2. You should know how to evaluate this: first, square the x, then multiply it by 2. Let's see how this is exactly like what we did before. You can see it like this: say that f stands for the action of squaring the x (it's a transformation—a morphism). And the other morphism g stands for multiplying it by 2. f=x2 g=2x If you're given the problem of determining what you have if you do g following f, you can write that like g∘f. Plugging in the actual equations, you are told to "multiply x by 2 following squaring x." Which is exactly what you did. If you were told to determine f∘g instead, you should be able to see that this tells you to "square x following multiplying x by 2." I.e., (2x)2. The symbol ∘ in fact tells you that you are composing one morphism with another (exactly how composition works is defined by the category), and the requirement that Af→Bg→C implies Ah→C such that h=g∘f is called compositionality. Every category has a rule of composition. Composition is the real action of any category. If you know the rule of composition, you know how the category works internally. (Think about the concept of the laws of physics, the view of the world as a chain of events such that we can predict our current state from the Big Bang. Understanding that rule of composition is the theory of everything, the holy grail of physics.) Now that we know what composition is, we can use it to rigorously define how the identity morphism "does nothing." Suppose you have an object A and the identity morphism 1A. Say you also have a morphism f:A→B. Since B is an object, it has an identity morphism as well, 1B. The identity morphism 1A is a morphism "on A"—what does that mean? It means that 1A goes from A to A. It goes nowhere, in effect—just what we'd expect of an identity morphism. So we have a morphism 1A:A→A. We also have a morphism f:A→B, as stated. Well, look at this! According to the rule of composition, we must have a morphism h:A→B such that h=f∘1A. But we already had a morphism going from A to B, f itself. The identity morphism does nothing, so it shouldn't have any effect on what we already had. This is precisely defined by the following condition: f∘1A=f. That is to say, doing f after doing 1A is the same as just doing f. Composing by 1 is just like multiplying by 1. And notice we also have 1B:B→B. And since we still have f:A→B, we again have a rule of composition saying that we have a morphism going from A to B that is equal to 1B∘f. What is this morphism? Again, 1B does nothing, meaning that 1B∘f=f. Basically, if you give a nut to a bird, and the nut is the nut, that's the same as just giving a nut to a bird. Similarly, if you give a nut to a bird, and the bird is the bird, that's just the same as giving a nut to a bird. Brilliant, I know, but sometimes math is about piling up trivialities until something profound emerges. There's one last rule we'd expect any model to obey. Let's take again the sequence of events where Alice pushes Bob, and Bob stumbles into Charles. Temporally, Alice pushed Bob before Bob ran into Charles. But logically, we should be able to study the events out of order: we should be able to study Bob's motion into Charles, and then look backwards in time to study Alice pushing Bob. (Just like how we can study Alice pushing Bob before we look backwards in time to see how the Big Bang pushed Alice into pushing Bob.) This ability to study the events out of order as long as ultimately put things together in the right order is called associativity. You might be familiar with this in terms of the associativity of multiplication, for example. Because (2×3)×4=2×(3×4),we say that multiplication is associative. Let's add David into the equation, so that we have a path like Af→Bg→Ci→D. We have a new composition Ai∘g∘f−−−→D. Associativity would tell us that (i∘g)∘f=i∘(g∘f). Indeed, in any category, the rule of composition must be associative. This should be interpreted as saying that we study at any individual interaction out of order, as long as we don't forget what the order actually was! (Why "study?" Because we're modeling models, and models are for studying reality.) You could think of associativity this way: Say you're planning a trip from North America to Europe to Asia. Associativity says that you can think about the trip from Europe to Asia before you think about the trip from North America to Europe so long as you remember to leave from North America when it's time to take the trip! And...we're done. We have the rules we'd expect any model to obey. And, not coincidentally, we now have the rules we'd expect any model of models to obey—the rules that define a category. Definition: a category C is a mathematical structure consisting of i) a bunch of objects written A,B,C, etc. ii) a bunch of morphisms written f,g,h, etc., which obey some rules. Rule 1: Every morphism goes "from" an object in the category C "to" an object in the category C. The "from" object is called the domain and the "to" object is called the codomain. (This should sound similar from learning about functions—functions are just a type of morphism; functions have a domain and codomain because all morphisms do.) For example, if you have a morphism f:A→B, then A is the domain of f and B is the codomain of f. Rule 2: The category C must have a rule of composition such that, if the domain of one morphism in C is the codomain of another morphism in C, then there is a way of composing the two morphisms to get a single morphism extending from the domain of the latter morphism to the codomain of the former morphism such that this new "composite" morphism is equal to the composition of the latter morphism following the former morphism. E.g., if you have morphisms f:A→B and g:B→C, then you necessarily have h:A→C such that h=g∘f. Rule 3: Every object in C has an identity morphism going from itself to itself that does nothing. For an arbitrary object A in C, the identity morphism is written 1A:A→A. The identity morphism "does nothing" in the sense that composing it with other morphisms is the same as just doing the other morphism. Rule 4: Composition is associative. Associativity means that if you have a chain of compositions i∘g∘f, then it is always true that (i∘g)∘f=i∘(g∘f). Why are there so many rules for morphisms, and none really for objects? We'll explore in a few posts from now the idea that categories are all about the morphisms. Basically, we'll explore the idea that an object is totally defined by what it does and what is done to it—i.e., all the morphisms it is a part of. Coming up next are examples of categories. This next post should hopefully both clarify the claim that category theory generalizes different fields of mathematics and also help make the concept of a category much more concrete. Afterward, we'll talk about a very important type of category, the category of sets and functions, and then a series of posts on both conceptual matters and enhancing our mathematical understanding of categories. Discuss ### Reflections on Premium Poker Tools Part 2: Deciding to call it quits 9 октября, 2019 - 07:17 Published on October 9, 2019 4:17 AM UTC For the backstory, see part one. There's a part of me that thinks that quitting should be an easy decision. A big part of me. The way I'm thinking about it, there are two reasons to continue working on this: 1. Swinging for a home run. If there's a chance for high upside. If not, there are other ideas I could be working on that do provide the upside I'm looking for. 2. Swinging for a single. If there's a high likelihood of making a few hundred thousand dollars. Something comparable to what I'd make working as a programmer. Swinging for a home run Throughout the time I was working on Premium Poker Tools, I had been assuming that the market was big enough to provide the high upside. Eg. I thought that 100k users and100/user was very possible.

• In being amongst the poker community, you get the impression that Flopzilla (market leading competitor) is ubiquitous. Everyone knows what it is. Everyone has it. Everyone needs it.
• The poker subreddit has over 100k subscribers. Educational YouTube videos get over 100k views. Some posts on TwoPlusTwo have over 1M views. The type of person to watch an educational YouTube video on poker is serious enough to want to have Flopzilla or a similar app, you would think. So maybe there are a few hundred thousand of these people in the world, and maybe I can capture 100k of them.
• On top of that, poker seems to be a game that is growing. And imagine if it is legalized in more states?
• At $100, the app pays for itself. And poker players are the type to understand +ROI decisions, you would think. • Poker players tend to be on the wealthy side. Older people with money. Younger people in high paying fields like tech. Younger people who are playing professionally. Well, most of them aren't actually wealthy, but it's especially important for them to invest in tools. • Maybe there's potential to be making hundreds of dollars per user. After all, that's still +ROI for a lot of people. But more recently, I've come to believe that that all just isn't true. • One reliable source said he'd estimate that the market leader, Flopzilla, has between 1,000 and 10,000 users. And they charge$25-35. Which would imply an upper extreme of just $300k or so. Before marketing costs. • Four other people in the industry I've spoken with all say the same thing, although they didn't want to get into specific numbers. • Looking at Flopzilla and the other competitors, they're mostly one man shows that are lacking in terms of design and business stuff. If the market were bigger and they were making millions of dollars, I'd expect them to have a team and for things to appear more professional. • It's becoming more and more clear to me that very few poker players actually sit down and study. And that's what my app is for: to help you when you sit down and study. People want to passively watch YouTube videos and be told what to do. Poker coaches talk all the time about how students who are paying them a lot of money are too lazy to do the homework they were assigned. • It's also becoming clear to me that poker players aren't these rational purchasers that I envisioned. They're not quick to open their wallets, and often don't think about whether or not it pays for itself. Even if the industry people I've spoke with are wrong about the market being eg. 10k users, I'd need them to be way off. Suppose I make$25/user after marketing costs. That would mean I'd need 400k users to get to $10M. Since I can't expect 100% market share, I'd really need the market to be something like 1M people, and that is a very long ways from the 10k upper extreme that people are telling me. Maybe the real market is 50k instead of 10k, and I get 20k of those 50k. At$25/user after marketing costs, that's only $500k. Swinging for a single If I do decide to quit working on Premium Poker Tools, my next step would be to take some time to work as a programmer and save up money before moving on to my next startup. But if I am close to the point of making a few hundred thousand dollars or something, then I may as well just keep working on PPT instead. I just don't think the chances are high enough that I'll get to first base, so to speak. Zooming out, I've been working on this app for 2 years and 3 months and only have two paid users. Furthermore, I've sort of exhausted a lot of the customer acquisition channels. Affiliate partnerships, paid advertising, YouTube, Twitter, blog, in person networking, offering it for free, participating in poker forums, refer a friend, offering to coach/study with people: none of it has worked. So looking at it from that perspective, it would be crazy to say, "The upside to make millions might not be there, but I'm on the verge of making a few hundred thousand, so I should see this through." Also, it's worth noting that throughout these past two plus years, I've continually thought to myself, "I'm just a month or two away". I clearly have a bias, and I need to adjust in the opposite direction. What if Like I was saying at the beginning: this should be an easy decision. The reason to work on this is if it has a big upside, or if I'm on the verge of "hitting a single". Neither of those things seem close to being true, so I should quit. That's all there is to it. I've went through this conversation with myself many times, and something just felt off. I was telling myself that the answer is obvious, but I still felt very hesitant to move forward with the decision to quit. It didn't feel right. Eventually I noticed what was holding me back: "what if". What if I'm wrong and there actually is a big market? What if I really am just a month or two away from hitting a single? The correct response to that is to note that you can never be 100% certain of anything, and just because there's some small chance I'm wrong doesn't mean I should keep working on this. You just have to make the best decisions you can given your uncertainty, not wait endlessly until you are certain. Of course. I know this. My Monkey Brain even knows this. It's something I've always been pretty good with. Let me try to do a better job explaining what has held me up. Imagine the following: I quit Premium Poker Tools, and proceed to do my thing working as a programer. Nine months later, I see a new poker app take off and make millions of dollars. I think, "What the hell! How did this happen!" I start looking into it. I find some financial website that shows Flopzilla making millions of dollars a year. I talk to some people who say, "Oh yeah, the market is huge." I'll tell you why this would be boundlessly frustrating to me. Because if I had just taken the time to do this research before quitting PPT, I could have been the one who succeeded. Because a trivial inconvenience is the thing that got in my way. I expect better of myself. If that happened, I'd be extremely disappointed in myself. I know I'm not perfect and there are things I'll miss. I just don't want to get caught in the above scenario where there are obvious things that I miss. Because of laziness. So then, the solution is pretty simple right? Just take the time right now to investigate all of these things and make sure I don't fall victim to trivial inconveniences. But somehow, that solution just doesn't seem to be working for me. I just keep worrying that there are things I'm missing. That there are more trivial inconveniences that I should look out for that I just don't want to because I'm lazy. I know that sounds weird; it's a hard feeling to explain. I feel tempted to just move on despite the voices in my head saying "this doesn't feel right". But if I do that, I know I'll torture myself with "what if" questions. Let's try writing about the "what ifs". Sorry if it gets too poker-specific. • What the people I spoke with are wrong and the market right now is actually really big? Even if that were true, it still seems unlikely that I'll succeed, given how little traction I've had. Anyway though, I feel confident that the market isn't big. What else can I do to increase this confidence? I could reach out to more people. I don't really want to do that though. I could learn more about researching your competitors. But I've done a bit of googling and it hasn't been too fruitful. One article said you can estimate based on the number of employees. I'm pretty sure they don't have employees, which would imply that the market is small. • What if I'm just being impatient and need to spend more time building an audience? Well, even if that were true, it would only be for a slice of a ~$300k pie. The amount of work/time doesn't seem like it's worth it for a small reward, especially given that it could easily just flop and not get me any traction. Also, I really enjoy writing blog posts and making YouTube videos, and it's something that I can do in my spare time if I decide to quit.
• What if I'm just one affiliate away from making this a success? Flopzilla does affiliate stuff, even with some A-listers, and they still only have 1k-10k users. So, as tempting as it is to think otherwise, it's pretty unlikely that one affiliate does the trick for me. Even if it does and it snowballs into more and more affiliates, we're still talking about a roughly $300k pie, and working with affiliates means giving up 30-50% in revenue share. • What if this new tool I want to build gets me over the hump? It's possible that it's a hit and it gets me, say, 2k users at$50/user or something. But that's not enough to justify continuing to work on this full time, given that it could also just flop. The upside would need to be like 10 or 20k users, and I really don't think it's that large. However, this tool I'm referring to is something that I plan on working on in my spare time regardless, and if it really takes off, I can always quit my job and pursue it if I feel it's appropriate.
• What if lowering the price and undercutting my competitors leads to a lot of traction? I highly doubt it. I had been offering it for free for a while, and it was only getting me like 100-200 users/month. I think it's more about exposure and having an audience than paying $20 instead of$25. But even if undercutting them is a success, how big a success are we talking? Maybe 5k users if I'm lucky. I can always try this in my spare time, and come back to Premium Poker Tools if it has a ton of traction all of a sudden.
• What if I can expand the size of the market by making it easy enough to uses for the masses? Right now, the masses are confused by and don't know how to use this sort of poker software. Even more serious players often fall into this bucket. It's always made me feel hopeful, because if I solve this problem, it could give me a big leg up. I have many ideas on how to do this. Some sort of workbook thing to go through. More handholding. A little gamification. But all of this would be a pretty big undertaking, and my intuition is that it wouldn't grow the market too significantly. And there's a lot of risk. It doesn't seem worth spending 6-12 months working on.
• What if I add stuff like spaced repetition to make studying more effective? I really don't see this being something that people will understand and value enough to pay a lot of money for. And I don't see it as something that will attract attention from the masses.
• What if I built something that helped you zoom out and find good exploitative spots? It'd be a big undertaking. Let's just say 6-12 months. It could easily just flop. That all makes it a bad candidate to "swing for a single" with. And, while I personally think it'd be awesome and that everyone else should agree with me and pay a lot of money for, I don't see it as having home run potential, unfortunately. It comes back to the fact that players don't want to study and do the work. Also, I think it'd be inherently complicated, which would make it harder to target the masses with.
• So many people have told me that they really like the app. With that sort of feedback, this should be making at least decent money, right? What if I'm just doing something horribly wrong? I don't know. I really don't. I guess a high quality product isn't enough, and for whatever reason I'm still far from the point of "hitting a single".

Even after writing all of this out, it still doesn't feel right. I feel like there are brilliant ideas that I had in the past and was excited about, that I put to the side and am failing to recall right now. I feel like I've built up to thee point where I'm slightly better than the competition, and that if I spend another six months or so, I'll really start to pull away and differentiate myself with a solid competitive advantage.

But all of these ideas I have. The things I'd do to pull away and differentiate myself. I don't actually think that any of them would really "grow the pie". They'd just give me a better shot of getting a slice of that ~$300k pie. And that just doesn't seem like it's enough. It could easily fail. Even if I build all of this cool stuff, people could just want to stick to the software they already know. Customer acquisition is hard. The things I'd be building would be much more of a vitamin than a painkiller, and so I don't think "build it and they'll come" would happen. So ~$200k reward let's say + 50% chance of success let's say (generously) + 12 months more work + a lot of stress and anxiety. And that's the optimistic equation. That just isn't good enough.

This is very, very difficult, but I've decided: it's time to call it quits.

Discuss

### Reflections on Premium Poker Tools: Part 1 - My Journey

9 октября, 2019 - 03:42
Published on October 9, 2019 12:42 AM UTC

I've been working on Premium Poker Tools for over two years now. I'm at a point where things aren't looking so hot, and I need to sit down and do some hard thinking. What better way to do that then to write on LessWrong!

Background

To understand my journey with Premium Poker Tools, I think you have to first understand a little bit about me and what I'm doing with my life. In short, I want to start startups, make a lot of money, and use the money to a) be financially independent, and b) do good for the world.

I spend about two years in college working on a startup that went nowhere. After that, I managed to get an entry level job as a web developer. I was a pretty bad at the time. I spent about 11 months at that job before getting fired. I quickly got another job, but got fired 5 months into that one. In both cases, it was a mix of a) me clearly not being a "company guy", and instead wanting to be done at 5pm so I could go off and do what's best for me, b) clashes with authority figures, and c) being a slow programmer.

After the second job, I moved to Vegas to be with my now girlfriend. I had about $85,000 saved up in the bank, and my plan was the following: 1) get better at programming, 2) freelance a little and save up some more money, 3) start a startup. And then from there repeat steps 2 and 3 (and maybe 1) until I succeed with a startup. After that, the plan would be some mix of 1) figuring out how to use the money for good, 2) starting or investing in more startups, and 3) enjoying myself. Self-studying computer science I spent my first year in Vegas self-studying computer science. It was alright. Idk. I feel like I should have came away from that year making a lot more progress than I did, but at the same time, I did learn a lot. In general, I've always planned on living the digital nomad life. At least until I find a place I like enough to want to settle down in. But I'm pretty sure that'll take some time, and that I'll end up spending time in a lot of different places. Anyway, I find myself living in Vegas. Poker is a thing in Vegas. I figure that I should spend some time on hobbies, and poker seems cool, so I feel like I should give poker a shot while I happen to be in Vegas. I won't have another opportunity like this. Or so my thought process was at the time. So, during that first year, I spend some of my time learning and playing poker. I have a lot to say about poker. I think it's a great game for rationalists and people into challenging themselves and self-improvement. I'll have to write about that in another post some time. As I learn about poker, I see these two apps come up over and over again: Flopzilla and Equilab. They're in books, and videos, and blog posts. People talk about them in poker forums. It seems like if you play poker, you really need to have these apps. They let you run simulations and see things like how often A3s beats JJ, and how often 76s flops a flush draw or straight draw. I really wanted to get these apps. But when I went to download them, I learned that you need a Windows OS, and I have a Mac. Damn! I was pretty bummed. Oh well. I proceeded to dabble around in poker, progress a lot more slowly than I would have liked, and spend most of my time studying computer science. Life was fine. A small project to dust off the rust After about a year, I felt like it was time to move on. I was ready to freelance as a web developer. However, I had been doing computer-science-y things for a year and my web dev skills got pretty rusty, so I wanted to work on a project first to dust off that rust. My skillset had been very JavaScript-centric, but I always thought the culture around Ruby was so cool, so I wanted to pursue that. But what project should I build? Hmmm. Oh! I know! I could build my own miniature version of Flopzilla and Equilab! It actually doesn't seem that hard! Seems like something I could do. And it seems like it'd be a lot of fun. Plus, I could use it to study poker. Awesome! And so, that's what I pursue. I work on it. I'm having a great time. Getting sucked in a little bit. I think it took about two months or so to have a decent version one. I wasn't really planning on spending that long on the project, but oh well, it's fine. I was really enjoying Ruby. So much nicer than JavaScript. I love how readable the code is. Then... the planning fallacy happens. Over, and over, and over, and over, and over, and over, and over again. I think that pretty much sums up my next two years. In some sense at least. Really getting sucked in I think to myself, huh, this is pretty cool. I actually built a miniature version of these popular apps. I wonder if this could be a thing. I get pretty excited. I know a guy (well, really an acquaintance of an acquaintance) who played poker professionally. I get a hold of him and ask if he thinks this app I'm building could be a thing. He says that Flopzilla and Equilab are definitely better, but that if this is free, well, idk, maybe it could be a thing. Good enough for me! Woooooooo hoooooooooooo!!! Let's goooooooooooooo!!! Says my Monkey Brain. Rational Self? He says: Meh, idk if this is a good opportunity or not, but there's a couple more features I want to build that shouldn't take too long. Idk, maybe a month or two. Let's just give it a go. I'm getting good experience, after all. And I feel like this could get some traction with a) Mac users and b) people who want something for free, at the very least. So yeah, let's just bang out these next couple of features and proceed from there. C'mon Rational Self. Will you ever stop falling for the planning fallacy? So yeah, at that point, I basically just get sucked in. I keep thinking, "only another month or two". But things always end up taking longer. At some point I deploy it so that it's live. When I test out the live version by running a simulation, I realize something: it's slow. Huh? Why is that? The development version has always been super fast. Why the discrepancy? Oh, I know: network latency! You see, I've been using my new friend Ruby on Rails. Which runs on the server. So when you click "Simulate", a request has to go out, over the internet, hit the server, run the simulations, then you have to wait for it to come all the way back to your laptop so that you can see the results. That back-and-forth from your laptop to the server takes time, and that time is why the live version appeared slow. Fuck. That's just totally not acceptable, from a user experience perspective. As if I have other people using this who I need to optimize for. So what do I do? I just have to rewrite the whole thing to run on the client side. Remember how this was just supposed to be a little project to dust off my web dev rust? I further justify the decision to rewrite it, because it'd be an opportunity for me to learn React. Previously I was an Angular guy, but in my year away from web dev React and Vue and these one way data flow frameworks became a thing. Data binding is bad now, apparently. So like, my skillset is pretty outdated if I don't know these new frameworks, right? I should take the time to learn them, right? Learning is Good, right? And besides, it won't take that long to rewrite it. The actual logic is all there in Ruby already. So, that's what I do. I rewrite the whole thing in React. I forget exactly how long it took, but it took some time. Guess what happens next? I decide to rewrite the whole thing in Vue! I actually remember how this went down in pretty good detail. There was a point where I wanted to do the equivalent of ng-repeat in React. I looked around, and couldn't find out how to do it. What?! That's such a basic thing. Why isn't there an easy way to do it. The React Way to do it just seemed really ugly. I was annoyed. Then I ended up taking a trip to New York to visit my family. There was a day where I was in the city for a doctors appointment. I was getting a bagel in the morning. I remember this. I sat down to eat my bagel. I opened my phone and was browsing Hacker News. I came across a post comparing VueJS to the other frameworks. I sat there and read the post as I ate my bagel. Monkey Brain: OMG. VUE IS AMAZING. YOU HAVE TO DROP EVERYTHING AND REWRITE THE WHOLE THING IN VUE. Rational Self: Yeah, I mean I guess that's not a bad idea. Learning is Good, after all. I'm not in a rush. Seems like a good thing to do. And so, I rewrite the whole thing in Vue. I have to say, I do like Vue a lot. Let me point something out though. At this point, I've been working on this project for... I don't know, maybe 6 months or something? A lot longer than I had initially planned. Is this a real business? And as I work on it, slowly but surely the idea starts to marinate in my head that this could be better that Flopzilla and Equilab. That this could be a real business. So yeah, I rebuild it in Vue. I'm loving Vue. I'm loving the app. I'm thinking that I might be able to build something better than Flopzilla and Equilab. (Let's just say "Flopzilla" moving forward instead of "Flopzilla and Equilab". Flopzilla is the more substantial of the two.) My memory for this point in the journey is a little bit fuzzy. I remember there being a good deal of time where I was in this no mans land where I wasn't sure if I wanted to pursue this as a business, but I did want to build "just one more feature". I basically just kept caving in and doing "just one more feature", while slowly thinking more and more, "Ok, I sorta think I could outdo Flopzilla. I sorta think this could be a business." At this time, I told myself that it's actually pretty low risk. Well, what I mean to say is that it's a pretty high-ceiling sort of pursuit. Like, I'm not sure if I could outdo Flopzilla, but I'm pretty sure I could get somewhere in the ballpark. Yeah, I felt pretty sure of that. And Flopzilla doesn't even work on Macs. And it costs$30. Mine does work on Macs, and I could make it free. So... worst case scenario I should be able to get a bunch of Mac users and people who want something for free. I know ads are tough. I'm not saying I could get rich off of a free app, but I should be able to get a lot of users, and it should do something for me. It should be a pretty high-ceiling sort of pursuit. It's hard to see it just totally being a flop if I make it available for free. And it'll only take another couple months. And who knows. Maybe it ends up being really awesome and I can charge for it and make good money!

So, I decide to pursue it. Well, I don't think "pursue" is the right word. For a long time, my thinking was, "I'm not sure if this is worth pursuing long term, but I do think it's worth spending the next few months building out a few more features, so let's just do that for now." Let me emphasize that. Even like a year into this project, it wasn't something that I actually.... in my mind... explicitly... decided to pursue long term. I just kept saying "another month or two, another month or two". And yet it had been like a year. I spent a year doing something I never actually decided to do, in some sense.

But after a full year, I didn't really have much to show for. That really started to get to me and make me anxious. How did this happen? How did I end up spending a year on something without ever deciding to do so?

To make it worse, it really conflicts with some part of my identity. I like to think of myself as "good at startups". I failed with my first startup, and I always thought to myself that next time around, I will be smart and rational and consider lots of ideas and really think it through. I'll come up with some awesome plan for something that is very likely to succeed. Or at least that has a super high EV.

But where am I now? That is totally not how things went down with this app I'm working on. There was no master plan at all. I just kept chugging along like some sort of semi-conscious monkey. Just another month or two. Just another month or two. Can't I open my eyes and think a little bit further down the line? Isn't imagining the future one of the things that makes humans human?

I know, I know. Humans are flawed and biased, and tons of people make these sorts of mistakes, and have these issues. But I expect better from myself. I know I can do better. Tsuyoku Naritai! I'm better than this. This is not what I'm capable of.

Struggling and moving slowly

But, I decide to chug along. Again, my memory feels pretty blurry. I forget exactly what my thought process was. I think it was that:

If I could go back a year, maybe this isn't the most optimal thing to work on. But I have all of this code. And I have this roadmap of features that I really want to build. I think it's at least worth taking 6 months or so to see it through, given where I'm at and how close I am.

As you can probably tell, around this time I really start to struggle emotionally. I felt pressure to succeed and make all this time I spent worthwhile. At least worth something in the end. I want to have something to show for.

I have thoughts that I'm 26 years old and I've always imagined myself way further along than I am now.

And I have this little voice in my head saying:

Hey Adam... it's been over a year... you don't have any users. This like totally goes against the whole lean startup thing, y'know.

And then I say in response:

I know, I know! But I really question whether the lean startup applies to this particular scenario. I know you want to avoid wasting time building things that people don't actually want. That's like the biggest startup sin there is. I know. But like, what am I supposed to do?My hypothesis is that once my app gets to the point where it's in the ballpark of Flopzilla or better, that people will want to use it. It takes time to get to that point. There isn't really a quick two week MVP I could build to test that hypothesis. I'm already trying to avoid building non-essential features, and focus on getting to that point as quickly as possible. So what am I supposed to do?If I released this and found that I had no users and no one wants this, what would that tell me? Just that people don't like this version of the app enough. Sure, ok. But the real question is whether they'll like the Ballpark Flopzilla version enough, or the Better Than Flopzillla version enough. My hypothesis is that they will, and releasing it now wouldn't invalidate that hypothesis. And I can't think of a way to test those hypotheses quickly. I think I just need to build it and see what happens.

I'm going to resist the temptation to respond to this right now. Right now I just want to tell the story. The story as it actually happened. But I do want to say that there were a lot of voices swimming around in my head questioning what I was doing.

Oh, one more thing. Throughout this time I'm also studying poker a little bit and trying to get good at it. It's going a lot more slowly than I wanted it to. I felt like an imposter not just as an entrepreneur, but as a poker player too.

I moved up the stakes online pretty slowly, but I was moving up. There was a point where I was stuck at $0.05/$0.10, but I did get past it. One thing that really ate at me though is that over my lifetime, I was down money at live $1/2. Live$1/2 is supposed to be the fishiest, easiest game ever. People love to talk about how incredibly bad $1/2 players are. And I'm down money in that game. And I've spent over a year building a poker app. And I'm supposed to be a smart person. Definitely surfaces feelings of being an imposter. At least for my Monkey Brain. Rational Self knew that it was just over a small sample size and I was in fact a profitable player at live$1/2. But it was still tough, and frustrating.

Such is life with poker though! That's part of what makes it such a great game! Go read some stories of the downswings that people go through.

Anyway, another big question that I debated with myself is whether or not to really release it and advertise it to the public. It had been over a year of development and I had a working, solid product, so when you zoom out, it seems pretty obvious that you should be getting it in front of as many users as possible. But... I didn't do that.

Here's why. I thought to myself, what exactly is the advantage of doing that? My big hypothesis is that Ballpark Flopzilla will succeed. I'm not there yet. Definitely at least a few months away. Sure, it's still possible that I get some users now, but who cares? I'll still get them in a few months anyway. It's not like I'd be losing them if I wait a few months.

Also, people like to say that the benefit to having users is that you can learn from them. But like, do I really need to release to the public for that? Aren't there other means to that end? Can't I just do traditional UX user research? Yeah, I think I can. Or so my previous self thought.

And so that's what I did. I tried to find people to sit with and go through the app with. I offered to buy lunch for anyone willing to do this with me, and tried to spread the word via online forums. Once in a while I'd find someone to do this with me, but to my surprise, not many people did. Weird.

Ok, but what's the downside, to releasing now, you ask? Well, I was very neurotic about bugs. I imagine the following thing happening:

I release some sort of beta version; people use it, they like it; but then someone realizes that my numbers are wrong. "It should be 81%, not 68%." They trash me in the poker forums; word gets out; I develop a bad reputation, people need to trust that the numbers I give them are right, that's absolutely essential!

So yeah, I really don't want that scenario happening. The risk of it happening seemed larger to me than the upside of getting a few users a few months earlier than I otherwise would.

And I'd always be telling myself: I'm not in a rush. So what if things take a little bit longer.

Lol! My Monkey Brain would be screaming at me that it needs some sort of validation. But then my Rational Self would say:

No, no, that's just an irrational feeling that I'm struggling to mitigate. We know it's silliness. Let's just ignore it.Validation!

Eventually, I did some tiny releases. And my Monkey Brain got some of that validation that it so desperately craved.

The big thing was posting to Reddit. I also posted to this popular poker forum called TwoPlusTwo, but I didn't get any responses there. But on Reddit, a lot of people told me the app was awesome, better than Flopzilla, etc, and thanked me for building it. I'd get a couple random emails from people telling this to me and thanking me. Man did this feel good. I really, really needed it.

Here's another high point. Maybe the biggest high point in my whole journey. This one I remember exactly where I was and how it went down.

My girlfriend had some friends over, and she likes to play loud music and have fun. Y'know, like normal people. I don't like that kind of stuff though. So I walked over to this awesome little hole in the wall place called Apache Taco Shop, hung out there, had a churro, and played some online poker.

I really shouldn't multitask like this, but I get bored when I play and pulled up YouTube to watch some poker videos in parallel to my playing. One of the videos was by a guy named Daniel Negreanu. He might be the most well known poker player in the world. People used to talk about him being one of the best in the world. Nowadays there's a cohort of younger more math-y people who are said to be better than Negreanu and his generation of players. But like, Negreanu is still fucking good.

I had like four tables open, I think. I have five windows on my screen. The left third is a YouTube video I have running. The right two thirds has four tables in a grid. Hands are popping in and out. Decisions to be made. My attention would drift to the YouTube video during the down moments in poker.

But as I had this video open, something caught my eye. In poker there's this thing called a hand matrix. It looks like this:

Normally in these sorts of videos the hand matrix comes from Flopzilla or Equilab. Sometimes it comes from some other random app. But this hand matrix was different.

It sorta looks like it's from my app. Premium Poker Tools. Could it be???????? NO WAY! I have Google Analytics and only get like 100 visitors a month. I only did a few small releases to Reddit. There's no way Daniel Negreanu is using my app and putting it in his poker videos!

But he did! It was my app! No doubt about it.

I swear, I almost cried I was so happy.

I called my girlfriend, told her, and ran home like a little baby.

I mean, this was big news (or so I thought at the time). He's one of the best players in the world. And a huge name. So, if he's using my app, I think I have to assume that it's good! If anyone has good taste it should be him. This is pretty strong evidence! Think about it. Imagine a world where Daniel Negreanu likes my app, but everyone else doesn't. That really doesn't make sense. If anything, there's going to be some niche of people who really like it. So yeah, this made me feel very confident that I had something legitimate on my hands here. That I wasn't crazy.

And the thing is, there's still so much room for improvement! I had so many ideas and features I wanted to pursue. If he likes it now, just wait until I make it even better! This has got to have a really high ceiling!

So, at this point, I'm all in with the app. I've finally decided that it's what I'm going to pursue long(ish) term. I forget exactly when my mind really was on board with that idea. I think it happened before the Negreanu thing, but by the time the Negreanu thing happened, my mind was definitely on board.

Finally starting to charge

At some point, I decide to finally start charging for the app, and start trying to actually get users. Until now, I just posted to Reddit a few times with updates, and the app was free.

There was one Reddit post in particular that inspired me to move forward. People were saying that it's kinda crazy that my app is available for free and isn't more popular. A few said that it's something that they would pay for. One person (in an earlier post actually) even asked how he could donate money to me. I replied with my PayPal info. Then a different person donated $10. (A little weird that the original person didn't.) That$10 I received, it was the first money I had made as an entrepreneur, and that's after three years of working on things (two on my first startup, and about one on this one). It felt good. But weirdly enough, it didn't feel as good as I thought it should feel, if that makes sense. And that thought made me feel a little weird, a little guilty.

Whatever though. Overall, I was pretty happy. Things were looking pretty good. People were saying that they love my app and would pay for it. And that's with the app at its current point. In six, twelve, eighteen months, it should be way better. And the competition seems like they aren't actively developing things anymore. So it seems like I should be able to outdo them and get a pretty big piece of the pie.

So, I proceed to finish up a few features, and then get to working on building user accounts and credit card processing stuff. Seems like something that is very standard, and should be a piece of cake for a good web developer, which I like to think of myself as. But it ended up being a big rabbit hole for me. I could have built a working version in a week or so maybe, but I wanted to do it right. I wanted to get security right, UX right, and just in general make sure I'm following most of the best practices. I never know how to explain why code takes longer than it "should", but I think this process ended up taking me at least a month, maybe longer. The fact that it took so long made me feel a little bit of imposter syndrome. Although when I came across Authentication Still Sucks, it mitigated a lot of that feeling actually. At the end of the day, I was pretty happy with the final result.

Then there were some other rabbit holes I fell in to. Mostly consisting of business stuff. How do I want to monetize? Charging seems like it makes the most sense, but I should at least educate myself on what the other options are, and explore them a bit. I do that and decide to go with charging. But then there are a million other questions. Do I go monthly? Annually? Something else? Multiple options? Multiple tiers? $X.99 or$X.00? Stuff like that. I ended up spending some time doing research (hat tip to Price Intelligently), talking to people, and trying to make good decisions about all of this stuff. Maybe a week or so? I forget. I ended up deciding to charge $5.00/month. Another small rabbit hole was looking at my options for credit card processing. I ended up going with Stripe, but I wanted to take some time to look at what the other options are, and what their costs and benefits are. Eventually, I finish all of the grunt work of setting up the code for charging people. It happened at a good time for me. I finished about a day before I had a two week trip to visit some college friends and go to one of their weddings. It was a good stopping point for me code-wise, and a good time for me to take a little break. Optimism and pessimism After coming back from my trip and letting the dust settle a little bit, I found that some things went really well, and some things went really poorly. I'll start with what went poorly: I wasn't getting any paid users. That was the big thing. This guy from Reddit was my first user. I Skyped with him twice. He's a programmer + entrepreneur too, so we had some nice chats. He emailed me saying that he wants to subscribe as a paid user to support me, but also wanted to be upfront that he wouldn't have signed up if he didn't know me. But that's because he doesn't play poker enough. He does think the app is better than the competition. Wonderful. Not exactly what I was looking for, but I'll take it! My second user is a guy that I used to study poker with via Skype every week. But then one day, he just stopped responding to me. I'd reach out every couple months, but he just wouldn't respond to me. Really weird. But yeah, he subscribed as a paid user. Third guy is someone random. Fourth guy is an IRL friend who I offered it to for free, but insisted on paying. So, all of that was nice, but it doesn't exactly scream product-market fit. Then there's the sign-ups. I offered a 14-day free trial. I was getting about one sign-up per day. It's something, but again, not really screaming product-market fit. So yeah, that was the bad news. Now for the good news. The good news is that a) I'd been getting good feedback on the app; people say it's really good, and b) I had a lot of affiliate leads. Let me explain what I mean by that. When I first started charging, I considered doing affiliate marketing, where I give someone a link, and anyone who signs up via that link, they get some cut of the revenue, eg. 25-50%. So like if John Doe signs up with their link, and I'm giving them 50% revenue share, since I'm charging$5.00/month, I would give them $2.50/month for every month John Doe remains a user. I initially decided to not do the affiliate stuff. I figured, why give up that 25-50% unnecessarily. Maybe the app takes off via word of mouth. I didn't actually expect that to happen, but I figured it was worth taking a stab. But after a month or so it became pretty clear that that wasn't going to happen, so I started to pursue affiliates. I was doing pretty good! I got in touch with a bunch of people who said that they're interested. Many of them were authors of books that I've read. That was really cool to me. Until that point, I thought of book authors as some sort of celebrity. Writing a book seemed like a big deal to me. And they were books that were relatively popular too. Now I don't feel that same awe for people who have written books. Anyway though, that's the part that was going well. I wasn't getting paid users organically. I wasn't even getting non-paid users organically (throughout the time the app was free, I'd only be getting like 200 users/month). But I was realizing how untrue "build it and they'll come" is, so it didn't worry me too much. In my mind, I had product validation. Enough people told me that the quality was really high. And my app's quality is only getting better, whereas the competition seemed stagnant. On top of that, I had all of these people saying that they were interested in working with me as an affiliate. People who were somebodies. Not really A-listers, but a solid mix of B-listers and C-listers. They seemed interested in working with me, and I felt pretty confident that once they start putting my app in their YouTube videos and blog posts, things would really ramp up for me. More optimism, more pessimism As time went on, the trends of the above section basically continued, but were amplified, I would say. Time went on and I still had those same four users. And I'd still be getting about one sign up/day. Actually, less. So that's the "more pessimism" part. Pretty bad that I'm not growing. The optimism part was that I had more people tell me they're interested in working with me as an affiliate. One was a legit A-lister. Let's talk about what happened with that. Red Chip I'm just going to go ahead and use real names. Screw it. During the World Series of Poker, I tried to get out there and meet people. I posted on a site called Red Chip Poker that I'd love to meet up and hang out with anyone who's in town. No one responded, but one person invited me to a private poker game that they're having. So I went. The game was super fun. By far the most challenging poker game I'd ever played in. I ended up having lunch with someone from the game. Partly just chatting, and partly looking for advice on my app. He was staying with a guy named Doug Hull, who's a cofounder at Red Chip, a pretty well known poker site that does videos, blog posts, forum, podcast, all of that stuff. He said I should get on their podcast. My first reaction was that Red Chip is an A-lister, and I wasn't ready to deal with A-listers. That was in my future plans, but I felt like I was too small to make my play at them. But, y'know what, screw it, let's do it. I told him yes, and went with it. He introduced me to Doug, and a few weeks later Doug invited me over to his house to chat. We chatted for like three hours. It was really nice for a lot of different reasons. One is just that there are only a handful of people in the world who build poker software, and he is one of them. So he understood things that no one else does, and it was nice to be understood in that sense. I felt a little awkward about the fact that Doug has a product that is sort of a competitor of mine. But he seemed to not feel that way at all. I spent some time walking him through my app. We had a really great conversation. He gave me a ton of useful feedback and advice, and I explained the thought process behind a lot of my decisions. It didn't feel adversarial at all. But it wasn't just feedback and advice. He wanted to work together. An A-lister! Wants to work with me! I guess now is the right time after all. What I was most interested in is something where they get me users for Premium Poker Tools. We talked about them putting a link on their site, promoting it, and getting some sort of revenue share. He was also interested in that, but was more interested in me building other tools for Red Chip. He does his programming in MatLab, and doesn't know any web stuff. So he can build stuff, but isn't great with UI, and isn't able to publish it to the web. On the other hand, I do have those skills. So he was really excited. He had a lot of ideas that he wanted to work together on. It felt like he "found his guy". We didn't get into the details too much, but it seemed like he wanted to co-own anything we built. I thought a lot about how I want to proceed after this. I was a little star-struck to be honest, because they are A-listers in the poker world in my mind. But at the same time, I wasn't really feeling a lot of Doug's plans to co-own stuff. 1) I really prefer to work alone. I like the autonomy. I want to be able to make decisions as I see fit, and not have to justify it to someone else. In particular, I didn't want to justify myself to Doug. I respect his thoughts, but I think there are a lot of times when he doesn't see things the same way that I do. Here's an example. I have a regular 2d graph on my app. Y'know, with x-y coordinates. When I showed it to him, he got excited and said that I should make it 3d or 4d, and how in his books he "one upped" me and has these 3d graphs. I seriously question whether users would be able to interpret those graphs and get value out of them, but Doug seemed to think it's just no big deal. Doug studied math in college and worked for Wolfram for a while, and seemed to not really understand how other people think. 2) I also felt like I didn't need him/them. All of the things he proposed are things that have been on my backlog. They're things I know how to build. I think the value he'd be providing would be product guidance, and marketing/exposure. I don't think I need the product guidance. And as for exposure, I could get that with other affiliates. I had a lot of people saying they were interested in working with me as an affiliate, so that seemed like something I could do. But despite these things, when you zoom out, it seems a little stupid for me to turn down an opportunity to work with them, given that they're A-listers and I'm sitting here with no traction after almost two years. But that's what I decided I wanted to do. Because I trusted my zoomed in logic. A couple days later, I had a Skype call with him and his partner James. It lasted a little bit over an hour. For the first half, I demoed the tools. James did most of the talking on this call, as opposed to Doug. He said he liked it! And he also had a bunch of useful feedback. The second half of the call, we talked more about business stuff and how to proceed. James was also interested in me building tools for Red Chip. He asked me how I felt about that. He threw out the idea of co-owning them. He also asked me how I felt about them sort of buying me out. Something along the lines of fixed price for PPT, and royalties + salary for future stuff. It was all very hand-wavy, but that was the gist. I said no. I said I wanted to own anything I do and really pursue PPT. I told him how I had these huge visions (which I exaggerated a little bit to be sales-y). That I thought it could make many, many millions of dollars. Eg. maybe 100,000 users, and a$200 average LTV. He sort of laughed at that. Not in a mean way at all, but he just said that doesn't seem very likely, and that the poker market just isn't that big at all. That 1,000 users and a $50 average LTV is a lot more plausible. I did update on this information, but not too much. It sounded like he was basing a lot of that on his numbers at Red Chip. But I saw the market for my app as way bigger. Just because they only have maybe 5k users doesn't mean that I am limited to that. In my mind, everyone needs software like mine to study with, whereas you don't really need to pay to be part of a training site. Reading books and consuming free content online like YouTube videos and blog posts will take you a long way. Yeah. IMO, those training sites really aren't necessary at all, given how much content is available for free/cheap. Also, I saw myself as capable of really skyrocketing past the competition and being the market leader, whereas there are a lot of poker training sites out there of similar quality to Red Chip. Which is to say there's a bigger market out there for me than for them. I didn't want to press him on this though. It didn't seem like it'd lead the conversation in a good direction. We finished up by talking about putting my app on Red Chip and doing the revenue share thing. He said they're interested in that. He said it'd be nice if I could have Red Chip users get some sort of Red Chip-specific branding on the site. I don't love that idea, but I said sure. He ended by asking if I'm open to having a different price. He said he thought$5/month is way too low and that you can't have a real business with that price. I said I'm totally totally open to that, and that I am sort of winging it with price and I really respect their industry knowledge and yeah, that we can definitely talk more about that. So that's how we left it off. That we'd talk again in a week or two.

I delayed a little bit though. For two reasons. 1) After getting feedback from Doug and James, there was so much I wanted to improve with the app before really putting it out there. 2) I wanted to figure out what a good revenue share amount would be. 25%? 50%? I needed time for a) reading and talking to people for advice, and b) to see if my other affiliate leads would materialize (starting to get concerned that they haven't yet), so that I have leverage (BATNA) going into the conversation. 3) Another interesting opportunity popped up for me. I'll talk about that in the next section.

Short deck

I co-organize the Indie Hackers Las Vegas meetup. One of the guys at the meetup made an introduction for me to a guy named Brad. Brad is a former poker pro who is working on some poker security and crypto things now. We had lunch. It was... interesting.

For one, he said that he really doesn't like the market for the sort of app I'm building. He said the market is pretty small. Same thing James said. He said that yes, there are millions of people who play poker, but that no one actually studies. Everyone likes to say that they study and sound smart and cool, but very few people actually do. And that poker players are cheap and don't want to fork over money. And that the type of app I have is outdated. Nowadays it's all about something called Game Theory Optimal plays, which basically is figuring out how you'd play against a perfect opponent, such that you can't get exploited.

But I only updated my beliefs slightly from this conversation. I have to update somewhat, but the things he was saying just seemed wrong. Sure, there are a lot of people who are lazy and don't study, and there are people who don't want to fork over the money. But there are also people who do study and who are happy to fork over the money. The pie is really big with millions of players in the world, and I only need a slice of it.

I told him about my estimates. The poker subreddit has over 100k subscribers. A lot of educational YouTube videos get 100k+ views. A lot of the smaller channels get tens of thousands of views. All of these people are serious enough about poker where they are spending their free time trying to learn the game. To me, that is a potential customer. If you're spending your free time trying to learn the game, you should be in the market for poker software. All of the books and articles I read say that you need to have poker software like mine. Flopzilla is a household name. Everyone knows about it.

And as for poker players not being willing to spend money, that didn't seem true. Poker players often are people with money. There are a lot of wealthy recreational players. There's a lot of older ones who are retired and have some money. Younger players are often in tech fields, or playing poker professionally. And poker is all about ROI. And apps like mine easily pay for themselves. So I would think that poker players wouldn't hesitate to invest in something with a positive ROI. I mean, not everyone, but again, the pie is big, so even if a minority do, that's still awesome.

And the idea that poker is all GTO now is an idea I'm pretty familiar with. Some people think this way. They're wrong. But more to the point, it's definitely not the case that everyone thinks this way. Especially small-to-medium stakes players. Most of them don't think that way.

So, for all of those reasons, I didn't update too hard off of his skepticism.

Anyway, that wasn't really what he was there to talk about. He wanted something from me.

There's a version of Texas Hold'em (what people usually mean when they say "poker") called Short Deck Texas Hold'em. Basically, you remove the cards 5, 4, 3 and 2 from the deck, and now flushes beat full houses. Brad plays high stakes short deck professionally. He says that everyone is good at regular poker (long deck), but people are still learning short deck, so there are (more) fish to make money from.

More to the point, there isn't any publicly available software for short deck. He said he's positive that some of the super high stakes guys hire people privately to build them software, but there's isn't anything available to the general public. So he wants me to adapt my app to make it work for short deck. He said that he would pay for it, and he knows a bunch of other people that would.

Furthermore, he was saying that he thinks it could be a great business opportunity for me. He said that there still aren't a lot of people who play short deck, so going mass market doesn't seem like it'd be the way to go, but the people who do play play super high stakes, and could be willing to pay a lot of money for the app. Like, hundreds of dollars a month. And possibly even thousands if I offer some sort of exclusive access, because these people really don't want their competitors getting access to it. He said that he is very, very well connected in the short deck poker world, knows everyone, etc. So we talked about how if I went through with this, we could do a thing where I give him a cut, maybe 25% or so, for anyone he gets to sign up.

I was hesitant to go off in a new direction, especially with the Red Chip thing being in the works, and having a bunch of other affiliates who said they'd work with me. But this did seem like a good opportunity. So I told him that I wanted to go home and see how long I think it'll take, and as long as it's nothing too crazy, I'm in. Surely enough, it didn't seem like too much work to adapt it for short deck, so I texted him and said I was in.

Things fall apart

At this point, I end up taking a week or two to finish up something I was working on, and then I got started working on the short deck stuff. That took 2-3 weeks I think.

About a week after talking to the Red Chip guys, I texted Doug and asked if we could wait another week. He said sure. Another week passes, and I text him. He doesn't respond. Whatever. I need a little bit of time anyway. I want to finish up this short deck stuff, and more importantly, I want to see if other affiliates materialize to give me more leverage against Red Chip, and I want to talk to some more friends for advice on the revenue share amount I should do. Initially I was thinking 50%. Some said that's way too high, and 20-25% is better. Some said that's too low, and 50% makes sense when you're getting off the ground. A lot of mixed messages, so I don't really know what I should do.

Moving back to short deck, over the 2-3 weeks I work on it, I'm in touch with Brad via text. He's responsive and nice. Going well. A few days in he texts me and says his partner is going to give me $500 when I finish, as an incentive for me. Great! Eventually I finish the short deck stuff. I have a call with Brad and his two business partners. First I demo it to them. They all love it. We ended up spending a lot of time on that call talking about my regular long deck app. One of his partners was pretty quiet; it was a different guy named Cy who did the talking along with Brad. Cy basically gave me the same spiel as Brad, about how much he hates the business. Whatever. More updating, but still nothing crazy, for the same reason as before. Cy has some different ideas as for how to monetize the short deck stuff. He wants to create a site, eg. shortdeckstrategy.com or something. The idea is to write some articles, put my app on the site, get visitors, and put an affiliate link to a poker site like Poker Kings, and make money when people sign up for Poker Kings. Brad's agreeing with Cy. So like, a totally different direction from what Brad and I talked about initially. Initially we talked about not going mass market, and instead selling to all of these high stakes players that Brad is supposedly so buddy-buddy with. Throughout that call, my head was spinning a little bit. There was a lot of fast talking. At the end of the call, we agreed that they would use the app for a little while, test it out and stuff, and we'd all mull things over regarding how to monetize. Then there's the question of how much they would pay me for the app. When Brad and I initially met, we threw around numbers like$50/month or $100/month. Nothing definite, just throwing numbers around. During this call when the topic came up, I just left the question to them and said "whatever you think is fair". Brad and Cy sort of looked at each other - to the extent that you can do that in a video call. Cy was hoping for$50/month. But Brad spoke up confidently and said let's do $100/month. Cy agreed, slightly reluctantly. And that was that. Oh, Brad also said that he knows so many people in the poker world and can introduce me to some people who could maybe be affiliates for me for my regular PPT app (as opposed to the short deck one). One in particular - Ryan Laplante. We left it off saying he would get me in touch with Ryan and hopefully some others. A few days go by. I haven't received any payment from them. I'm sure I'll get it though. I don't want to be pushy if we're going to be business partners. I get my$100/month from each of them. But I don't get the $500. Another few days go by, and I ask Brad about it. He's sort of my point-man. I won't narrate the exact text history, but it was really weird. I also followed up on the intro to Ryan Laplante and others. And I asked him for Google ad advice, because he said he's good at that. A few other things I think. Sometimes he'd just straight up ignore me. Sometimes I'd get a short and cryptic response. For the$500, sometimes he'd ignore it, sometimes he said he'd ask Cy and then not get back to me.

I start to get annoyed at this. I don't want to work with someone who isn't responsive. And I definitely don't want to work with someone who promises me money and then doesn't give it to me.

Well, actually I'm not sure about this. I debate it internally with myself. Maybe I should just suck it up and deal with it, because it'd be too much to pass up the larger opportunity.

Let's jump back to the Red Chip thread. Eventually Doug gets back to me (he hadn't responded to my last text). He apologizes. I say no worries, let's schedule a time to chat again. His response to this was really weird. He said sure, want to come over to my house, or meet at the bar. I'm confused. James, his partner, the one who seems to be more in charge of business stuff, lives in Tennessee, and we're in Vegas. I say I want to talk about money and our next steps, and I think James should be there for that. His response to this really surprised and frustrated me. It was that James is a "hard guy to reach".

What?! This is your business partner. And we're in the process of working together. What's going on? It was as if our previous two conversations — the ones we spent 4+ hours on — didn't happen.

Eventually he says that if it involves money it needs to be in writing anyway, so why don't I write up my proposal and email it to them, and then we can talk afterwards. I don't love that idea, because to me this is a fluid conversation that involves back-and-forth. But sure. I email them.

I decide to totally err on the side of being generous. I was considering offering 25%, but I decide to go with 50% revenue share. So I say that in my email. I also say that I want to go with $5/month for now, the idea being that I don't think I could reach the masses — eg. 100,000 users — at a higher price. I said that I want to at least give that a shot. And if that doesn't work, then it means my users will be more niche and more serious players, in which case I agree with them that it makes sense to charge more. Doug responds in this formal tone and says that they don't want to work with me. What?!!!!! What on earth just happened. We had everything set up. I thought we were just negotiating the revenue share amount, and figuring out the next steps moving forward. I'm so confused! He mentions very briefly that they don't want to work with me at the low revenue per user. That just makes no sense to me. 1) Aggregate is what matters, not per user. There aren't really any marginal costs per user. Customer support would be the big one, but I'd be handling that, not them. 2) I was clear that I'd only want to pursue the low price if it gets tons of users. Revenue is what matters. 3) At the end of the day, I am flexible. It'd be nice if we could at least have a conversation about it, given that we had already spent 4+ hours talking with each other and were so close to working together. So I respond and ask if he would mind explaining his thinking. He says they don't like the low price per user, because of the "fixed costs of partnership". That makes no sense to me. I didn't think the plan was to do any fancy lawyer stuff. And all I'd be looking for from them is to put the link on their site, and hopefully include it in their videos and blog posts as they see fit. So I don't know what just happened here. I would expect to at least have a conversation about it. His response was very short and formal, again, and he clearly wasn't interested in talking more with me, so I gave up. But I had two small parting requests. The initial thing we talked about was me being on their podcast, so I asked if I could do that. He lied to me and said they only do the podcast "sporadically" now. It used to be once a week, now it's once every two weeks. The second thing was asking if they could include PPT in their Best Poker Software article, where they list out all of the options people have in terms of poker software. He said they only revisit that article once a year. That was a total disaster. Let's get back to the short deck thing. Actually, before that, I want to note that none of my affiliate stuff has materialized. All of the people who said that they're interested in working with me just haven't been responsive at all. I spent time trying to find other people to work with me. Vlogers, Twitch streamers, coaches, whoever I could find. I sent out a bunch of emails and messages, but nothing really materialized from that. Two leads. More on that later. Also, I haven't gotten any new paid users. And sign ups has slowed to a crawl. Around this time I tried to learn about marketing and stuff. Trying to figure out what else I could be doing to acquire users. I spent a lot of time on the code, and wanted to put on my marketing hat. I went through Julian Shapiro's Growth Guide, which I loved. I appeared on two podcasts, but they were both small, and didn't seem to drive any traffic, let alone sign ups or paid users. I set up paid ads on Google, Bing, Reddit, and TwoPlusTwo (a popular poker site). They didn't get me any paid users though. Also around this time, I have lunch with a guy in the industry who knows his stuff. He tells me that he'd estimate Flopzilla to have between 1k and 10k purchases, at$25-35/user, which would mean a maximum of $350k. So yeah, everything is failing for me right now. Let's get back to the short deck thing. At some point I think through what they were saying on the call. At the time my head was spinning, but once the dust settled, going the mass market + affiliate link route did seem like a good way to monetize. So I write up an email to them explaining that I agree with that approach and what my thought process was. I ended by asking them what they think, and if we could set up a time to talk again. A week or two passes, and no one responds to that email. I start getting a really bad vibe from the short deck guys. I start pushing for the$500 some more, and still don't get it. So I start thinking about it, and I decide that I don't think I should work with them. a) Our relationship has soured given how they've been treating me, and b) I really question whether I even need them. Their idea was to have that shortdeckstrategy.com site, make money with affiliate links, and put my app on there for free to drive traffic (I would upsell some tools) along with them writing a few articles. But I feel like the free software is the big driver of traffic, not the articles they would write, so why don't I just put the short deck thing out there for free and put my own affiliate link on it? Why do I need them? The reason would be if they are so well connected and amazing that they figure out how to drive a bunch of traffic to the site. Something I clearly struggle to be able to do. But if they're so well connected, why are they the only three signed up for the short deck app write now? And why hasn't Brad been able to introduce me to anyone, like he said he would? It just felt fishy. Plus, the benefit would need to be quite large for me to justify working with people I don't like, and the benefit did not seem that large at all.

So I finally manage to get a lunch scheduled with Brad. Previously we had scheduled times to talk, but he flaked on me, twice I think. Before this lunch I really pushed for the $500. He finally responded and said that Cy thought he would be getting the app for$50/month, not $100/month, and that's why he hasn't paid me the$500. Ugh! That's just wrong on so many levels. I respond to Brad saying that I think that's wrong and that I am expecting to receive that money.

I decide going into it that I'm basically done with them, and only would possibly work with Brad if a) I get my $500, and b) he wants to go the route of getting people to sign up for the short deck app in exchange for 25% revenue share, not their shortdeckstrategy.com thing. So, we have lunch. I get to the restaurant, and I see Cy sitting there early before Brad gets there. I sit down and say hi. He immediately gives me$500. Cool. Got that over with. That was actually the big thing I wanted to accomplish by having lunch with... well, I didn't know Cy was going to be there actually, but I was going to talk to Brad about it. But yeah, happy I got my money. No apology from either of them though, which isn't good.

During the lunch Cy vents a little more about how he doesn't think my long deck app will succeed. Whatever. I'm sorta just passing time. At some point I plan on saying how I don't want to work with them on the shortdeckstrategy.com thing. I don't want to say it too early on though, because then what are we supposed to do for the rest of the lunch?

But eventually I get to it and tell them I don't see that it makes sense for us to work together on that, and that I would prefer to do it on my own. Brad is pretty taken aback. Looks like he didn't know how to respond. He says nicely that if I'm going to do it on my own, then I'd be competing with them, and it wouldn't make sense for us to work together. I say I agree. This is towards the end of the lunch, and I make a last attempt to see if Brad wants to do the thing where he gets 25% for anyone who he gets to sign up. He says no. Cy mentions that he's "ran out of energy". We leave, and that's that.

At the end of all this, I finally start questioning whether this is something I should continue to work on.

Things continue to fall apart

At this point there are three big affiliates who said they were interested in working with me that I have been hoping to work with.

The first is a guy who I was introduced to through a friend, and has a popular vlog. After the friend made the intro via text, we exchanged emails and he said he's interested and we'd set up a time to talk. Months had passed, and I had reached out via email maybe six or seven times to check in, and never got a response. Then one day I'm at the casino playing some poker, and I see him in person at the water cooler, so I walk over. He apologizes and says that he's just so busy and has so many of these offers, and that he doesn't even like promoting other products in the first place. I make a small push, saying that it's a great product and it'd be super low-effort for him: just put a link in the description, but he didn't seem interested, so I backed off.

The second is a guy who I really like and have been in touch with for about six months. He just keeps saying that he wants to work together but is too busy at the moment and needs to finish up a few things before we dive into it. I've given up on that at this point.

The third is one is really frustrating. I appeared on his podcast, and it went pretty well. After the podcast interview, I talked with him about working together as an affiliate. I offered 50% revenue share if he's going to be an active partner, or 25% if passive. For active, I just want him to produce maybe 2-3 blog posts or videos a month. For passive, he could just throw the link wherever and hope people click in. He said he's in for the active deal. Things were very clear, and it looked like we were ready to move forward. Then, well, he just disappears. A month and a half has passed and he hasn't responded to me.

So at this point, it seems that I've really ran out of gas as far as partnering with affiliates goes.

I try some other things though. One thing I try is changing the pricing from $5/month to$25 as a one time fee for the basic plan, or $75 for the premium one. That didn't do anything for me though. But as I am doing these new prices, I realize that there's a bug. People who are signed in, when they try to go to the purchase page, it says they are unauthorized. Sure enough I see the typo in my router file. Then I learn that my little chat widget, people have been trying to use it to tell me that there's the bug for over a month! And I haven't responded to any of them! The widget is supposed to forward me an email of the messages, but I haven't been receiving those email notifications due to some weird incompatibility issue with my mail provider, so I just have been assuming this whole time that everything is fine. What a nightmare. People try to pay me. Bug prevents them from paying. People try to let me know about the bug. They don't get a response. Anyway, I also try to do the short deck thing on my own. I made one tool available for free and another for$150 as an upsell. I'm going to put an affiliate link on the free tool to see if I could make some money that way, but I haven't been getting any pageviews for it.

I spent some time working on a blog, and posted a few articles to poker forums, but that hasn't gotten me any traction either.

Throughout all of this, I'm moving closer and closer to calling it quits. I'll talk about that in the next post.

Discuss

9 октября, 2019 - 00:31
Published on October 8, 2019 9:31 PM UTC

There has been considerable debate over whether development in AI will experience a discontinuity, or whether it will follow a more continuous growth curve. Given the lack of consensus and the confusing, diverse terminology, it is natural to hypothesize that much of the debate is due to simple misunderstandings. Here, I seek to dissolve some misconceptions about the continuous perspective, based mostly on how I have seen people misinterpret it in my own experience.

First, we need to know what I even mean by continuous takeoff. When I say it, I mean a scenario where the development of competent, powerful AI follows a trajectory that is roughly in line with what we would have expected by extrapolating from past progress. That is, there is no point at which a single project lunges forward in development and creates an AI that is much more competent than any other project before it. This leads to the first misconception,

Continuous doesn't necessarily mean slow

The position I am calling "continuous" has been called a number of different names over the years. Many refer to it as "slow" or "soft." I think continuous is preferable to these terms because it focuses attention on the strategically relevant part of the question. It seems to matter less what the actual clock-time is from AGI to superintelligence, and instead matters more if there are will be single projects who break previous technological trends and gain capabilities that are highly unusual relative to the past.

Moreover, there are examples of rapid technological developments that I consider to be continuous. As an example, consider GANs. In 2014, GANs were used to generate low quality black-and-white photos of human faces. By late 2018, they were used to create nearly-photorealistic images of human faces.

Yet, at no point during this development did any project leap forward by a huge margin. Instead, every paper built upon the last one by making minor improvements and increasing the compute involved. Since these minor improvements nonetheless happened rapidly, the result is that the GANs followed a fast development relative to the lifetimes of humans.

Extrapolating from this progress, we can assume that GAN video generation will follow a similar trajectory, starting with simple low resolution clips, and gradually transitioning to the creation of HD videos. What would be unusual is if someone right now in late 2019 produces some HD videos using GANs.

Large power differentials can still happen in a continuous takeoff

Power differentials between nations, communities, and people are not unusual in the course of history. Therefore, the existence of a deep power differential wrought by AI would not automatically imply that a discontinuity has occurred.

In a continuous takeoff, a single nation or corporation might still pull ahead in AI development by a big margin and use this to their strategic advantage. To see how, consider how technology in the industrial revolution was used by western European nations to conquer much of the world.

Nations rich enough to produce the rifle maintained a large strategic advantage over those unable to. Despite this, the rifle did not experience any surprising developments which catapulted it to extreme usefulness, as far as I can tell. Instead, sharpshooting became gradually more accurate, with each decade producing slightly better rifles.

Continuous takeoff doesn't require believing that ems will come first

This misconception seems to mostly be a historical remnant of the Hanson-Yudkowsky AI-Foom debate. In the old days, there weren't many people actively criticizing foom. So, if you disagreed with foom, it was probably because you were sympathetic to Hanson's views.

There are now many people who disagree with foom who don't take Hanson's side. Paul Christiano and AI Impacts appear somewhat at the forefront of this new view.

Recursive self-improvement is compatible with continuous takeoff

In my experience, recursive self improvement is one of the main reasons cited for why we should expect a discontinuity. The validity of this argument is far from simple, but needless to say: folks who subscribe to continuous takeoff aren't simply ignoring it.

Consider I.J. Good's initial elaboration of recursive self improvement,

Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion’, and the intelligence of man would be left far behind.

The obvious interpretation from the continuous perspective is that by the time we have an ultraintelligent machine, we'll already have a not-quite-ultraintelligent machine. Therefore, the advantage that an ultraintelligent machine will have over the collective of humanity + machines will be modest.

It is sometimes argued that even if this advantage is modest, the growth curves will be exponential, and therefore a slight advantage right now will compound to become a large advantage over a long enough period of time. However, this argument by itself is not an argument against a continuous takeoff.

Exponential growth curves are common for macroeconomic growth, and therefore this argument should apply equally to any system which experiences a positive feedback loop. Furthermore, large strategic advantages do not automatically constitute a discontinuity since they can still happen even if no project surges forward suddenly.

Continuous takeoff is relevant to AI alignment

The misconception here is something along the lines of, "Well, we might not be able to agree about AI takeoff, but at least we can agree that AI safety is extremely valuable in either case." Unfortunately, the usefulness of many approaches to AI alignment appears to hinge quite a bit on continuous takeoff.

Consider the question of whether an AGI would defect during testing. The argument goes that an AI will have an instrumental reason to pretend to be aligned while weak, and then enter a treacherous turn when it is safe from modification. If this phenomenon ever occurs, there are two distinct approaches we can take to minimize potential harm.

First, we could apply extreme caution and try to ensure that no system will ever lie about its intentions. Second, we could more-or-less deal with systems which defect as they arise. For instance, during deployment we could notice that some systems are optimizing something different than what we intended during testing, and therefore we shut them down.

The first approach is preferred if you think that there will be a rapid capability gain relative the rest of civilization. If we deploy an AI and it suddenly catapults to exceptional competence, then we don't really have a choice other than to get its values right the first time.

On the other hand, under a continuous takeoff, the second approach seems more promising. Each individual system won't by themselves carry more power than the sum of projects before it. Instead, AIs will only be slightly better than the ones that came before it, including any AIs we are using to monitor the newer ones. Therefore, to the extent that the second approach carries a risk, it will probably look less like a sudden world domination and will look more like a bad product rollout, in line with say, the release of Windows Vista.

Now, obviously there are important differences between current technological products and future AGIs. Still, the general strategy of "dealing with things as they come up" is much more viable under continuous takeoff. Therefore, if a continuous takeoff is more likely, we should focus our attention on questions which fundamentally can't be solved as they come up. This is a departure from the way that many have framed AI alignment in the past.

Discuss

### Formal Metaethics and Metasemantics for AI Alignment

9 октября, 2019 - 00:02
Published on October 8, 2019 5:37 AM UTC

A Brief Introduction to MetaEthical.AI

tl;dr: AIXI for Friendliness

[Crossposted from my new blog.]

Abstract: We construct a fully technical ethical goal function for AI by directly tackling the philosophical problems of metaethics and mental content. To simplify our reduction of these philosophical challenges into “merely” engineering ones, we suppose that unlimited computation and a complete low-level causal model of the world and the adult human brains in it are available.

Given such a model, the AI attributes beliefs and values to a brain in two stages. First, it identifies the syntax of a brain’s mental content by selecting a decision algorithm which is i) isomorphic to the brain’s causal processes and ii) best compresses its behavior while iii) maximizing charity. The semantics of that content then consists first in sense data that primitively refer to their own occurrence and then in logical and causal structural combinations of such content.

The resulting decision algorithm can capture how we decide what to do, but it can also identify the ethical factors that we seek to determine when we decide what to value or even how to decide. Unfolding the implications of those factors, we arrive at what we should do. All together, this allows us to imbue the AI with the necessary concepts to determine and do what we should program it to do.

See the open source code and commentary at www.metaethical.aiMotivation

Researchers like Eliezer Yudkowsky and Nick Bostrom, among others, have argued for the urgent need to develop a rigorous framework for ensuring that smarter-than-human intelligence will be beneficial for humanity. I think of them and the community around their respective nonprofits, the Machine Intelligence Research Institute and the Oxford Future of Humanity Institute, as tending to share a cluster of views, including the following:

• We should have a wide probability distribution over when human-level AI will be developed. If we go by expert predictions, most predict it will arrive within this century. A cautionary approach should prepare for even more rapid timelines.
• Once AI that is at least as intelligent as its human creators has been developed, there is a positive feedback loop in which it can take over the task of improving its own intelligence, quickly resulting in a superintelligence vastly greater than that of humans.
• A silicon-based intelligence will have many natural advantages that would further compound this process, e.g. ease of faithful replication, readily available additional hardware, already million-fold increase in serial computational speed relative to biological neurons, long-term exponential Moore’s law trend as well as realistic plans and models for its continuation.
• Virtually any sufficiently advanced intelligence will converge upon certain instrumental goals to persist and acquire more resources and power, if only to better serve whatever intrinsic goals it may have.
• There is no automatic guarantee that greater intelligence coincides with better ethics. There is also tremendous economic incentive to develop ever smarter AI but not necessarily to make it safer or beneficial in the long run. If anything, each private party’s incentive may be to cut corners on safety to get to market quicker.
• Many naive approaches to aligning AI with our values fail. Human values have a great deal of hidden complexity and missing just one dimension can lead to very undesirable outcomes. Therefore, a metaethical approach seems to be more promising than hoping to capture all ethical principles at the object level.

My own inquiry into metaethics began long before these ideas were written. In fact, I had even reached a point where I felt I could explain my reductionist metaethics to other philosophers. But having followed the development of the AI safety literature with great interest, I felt a renewed sense of purpose and urgency. It seemed we’ll need to not only solve perennial philosophical problems but do so with sufficient precision to make a computer understand them. What is more, it looked like we were in a race to accomplish it all before the arguably exponential advancement in AI crossed some unknown threshold.

Having mentioned all this, I will not be arguing for the above claims here. And while this forms my primary motivation, I actually don’t think agreement with any of them is necessary to appreciate the metaethics and metasemantics I develop here. I have spent enough time in academic philosophy to appreciate such theories in the theoretical spirit in which they have often been developed. Formulating them in code as I have done could be seen as just a notational variant to the more conventional expression of certain rigorous philosophical theories in mathematical logic. Doing so helps us avoid misleading vagueness and ambiguity and ensures maximal precision in our thinking and communicating, all of which can be appreciated without regard to any practical applications.

Still, I hope many of you have already or will soon come to appreciate some of the backdrop of this MIRIFHI cluster of views. It’s said that necessity is the mother of invention. It has certainly driven me to be more ambitious and aim for higher precision than I thought possible in philosophy. To have any hope of success, I realized I would need to delve into mathematics and computer science and bridge the conceptual divide. In doing so, I was excited to discover new inspiration and draw connections I doubt I would have made otherwise. And taking on an engineering mindset, I found myself pruning search trees that initially sounded appealing but turned out not to be fruitful, while finding new appreciation for theories that shed new light and enabled further technical progress.

While many areas of philosophy can benefit from a more technical mindset, I think conversely many in computer science or other mathematical fields may be too eager to apply whatever technical tools they may currently have at their disposal without pausing to ponder whether a problem is still at a philosophical stage in which important conceptual advancements must first take place. Perhaps these advancements have even been made already in academic philosophy but they are not aware of them, while the philosophers in turn are not aware of how to formalize them.

What follows is a mixture of original contributions to philosophical problems, some standard or not-so-standard components borrowed from across computer science and philosophy, and novel ways of weaving them all together. Throughout it all, I have tried my best to balance faithfulness to the subtleties of philosophical reality, the rigor of formalizing these theories, the urgency of making and communicating this progress, and the practicalities of engineering an initial prototype of a wildly ambitious project.

While I don’t necessarily trust our civilization to get philosophy right, I think it is quite good at making progress on well-defined technical problems. I hope I have largely succeeded in turning the philosophical problems of getting an AI to understand and share our values into an engineering problem — and hopefully one we can solve in time.

See the open source code and commentary at www.metaethical.aiAn Apology

In an ideal world, I would have accomplished the above while explaining and justifying each philosophical step up to the standards of contemporary analytic philosophy and relating them to the current and historical literature. Moreover, on the technical side, the mathematical formulas would be written and typeset in beautiful LaTeX with ample diagrams and gentle tutorials.

Or you know, I could have at least written it in English. Instead, I chose to write it in a little known programming language called setlX (although I’ve since interspersed the code with considerable philosophical comments). My choice at the time and perhaps even now, was on the one hand, to struggle with writing math with limited experience or institutional support. Or on the other hand, I could leverage my long experience and intuition with programming to write essentially the same content in a language with clear semantics in set theory — the classic lingua franca of mathematics. On top of that, I’d have a compiler to check for bugs and an interactive console serving as a concrete interface by which to manipulate very abstract objects and functions.

In further defense of setlX, I find it to be a very elegant and powerful language. Its relatively few primitives are sufficient to concisely construct complex data and algorithms while being small enough to pick up fairly quickly if you have some experience with programming, mathematics or logic. Not surprisingly, writing in it feels like you’re programming close to the mathematical essence.

Despite its initial unfamiliarity, I hope you will give setlX, my code and my commentary a chance. Even if you are not technically inclined, I expect that with a little patience, a synopsis can be gleaned from the comments — the most important of which I’ve gathered into a Key Concepts outline beside the code. Other than that, I have not imposed much of an ordering but tried to enable a more free exploration by hyperlinking procedure calls to that procedure’s definition, which often has at least a short explanatory comment.

Where I have left important explanations and justifications sparse, I’ve tried to include links to those of others who have likely done a better job than I would have. I wish I could have done more but I have mainly been optimizing for solving the problem rather than communicating the solution.

See the open source code and commentary at www.metaethical.ai

Discuss

### Book summary: Unlocking the Emotional Brain

8 октября, 2019 - 22:11
Published on October 8, 2019 7:11 PM UTC

If the thesis in Unlocking the Emotional Brain (UtEB) is even half-right, it may be one of the most important books that I have read. Written by the psychotherapists Bruce Ecker, Robin Ticic and Laurel Hulley, it claims to offer a neuroscience-grounded, comprehensive model of how effective therapy works. In so doing, it also happens to formulate its theory in terms of belief updating, helping explain how the brain models the world and what kinds of techniques allow us to actually change our minds. Furthermore, if UtEB is correct, it also explains why rationalist techniques such as Internal Double Crux [1 2 3] work.

UtEB’s premise is that much if not most of our behavior is driven by emotional learning. Intense emotions generate unconscious predictive models of how the world functions and what caused those emotions to occur. The brain then uses those models to guide our future behavior. Emotional issues and seemingly irrational behaviors are generated from implicit world-models (schemas) which have been formed in response to various external challenges. Each schema contains memories relating to times when the challenge has been encountered and mental structures describing both the problem and a solution to it.

According to the authors, the key for updating such schemas involves a process of memory reconsolidation, originally identified in neuroscience. The emotional brain’s learnings are usually locked and not modifiable. However, once an emotional schema is activated, it is possible to simultaneously bring into awareness knowledge contradicting the active schema. When this happens, the information contained in the schema can be overwritten by the new knowledge.

While I am not convinced that the authors are entirely right, many of the book’s claims definitely feel like they are pointing in the right direction. I will discuss some of my caveats and reservations after summarizing some of the book’s claims in general.

Emotional learning

In UtEB’s model, emotional learning forms the foundation of much of our behavior. It sets our basic understanding about what situations are safe or unsafe, desirable or undesirable. The authors do not quite say it explicitly, but the general feeling I get is that the subcortical emotional processes set many of the priorities for what we want to achieve, with higher cognitive functions then trying to figure out how to achieve it - often remaining unaware of what exactly they are doing.

UtEB’s first detailed example of an emotional schema comes from the case study of a man in his thirties they call Richard. He had been consistently successful and admired at work, but still suffered from serious self-doubt and low confidence at his job. On occasions such as daily technical meetings, when he considered saying something, he experienced thoughts including “Who am I to think I know what’s right?”, “This could be wrong” and “Watch out - don’t go out on a limb”. These prevented him from expressing any opinions.

From the point of view of the authors, these thoughts have a definite cause - Richard has “emotional learnings according to which it is adaptively necessary to go into negative thoughts and feelings towards [himself].” The self-doubts are a strategy which his emotional brain has generated for solving some particular problem.

Richard’s therapist guided Richard to imagine what it would feel like if he was at one of his work meetings, made useful comments, and felt confident in his knowledge while doing so. This was intended to elicit information about what Richard’s emotional brain predicted would happen if it failed to maintain the strategy of self-doubt, allowing Richard to act confidently. The book includes the following transcript of what happened:

Richard: Now I’m feeling really uncomfortable, but-it’s in a different way.Therapist: OK, let yourself feel it - this different discomfort. [Pause.] See if any words come along with this uncomfortable feeling.Richard: [Pause.] Now they hate me.Therapist: “Now they hate me.” Good. Keep going: See if this really uncomfortable feeling can also tell you why they hate you now.Richard: [Pause.] Hnh. Wow. It’s because… now I’m… an arrogant asshole… like my father… a totally self-centered, totally insensitive know-it-all.Therapist: Do you mean that having a feeling of confident as you speak turns you into an arrogant asshole, like Dad?Richard: Yeah, exactly. Wow.Therapist: And how do you feel about being like him in this way?Richard: It’s horrible! It’s what I’ve always vowed not to be!

Richard had experienced his father as being assertive as well as obnoxious and hated. His emotional brain had identified this as a failure mode to be avoided: if you are assertive, then you are obnoxious and will be hated. The solution was to generate feelings of doubt so as to stop him from being too confident. This caused him suffering, but the prediction of his emotional brain was that acting otherwise would produce even worse suffering, as being hated would be a terrible fate.

UtEB describes Richard as having had the following kind of unconscious schema:

Perceptual, emotional and somatic memory of original experiences: his suffering from his father’s heavily dominating, hyper-confident self-expression, plus related suffering from unmet needs for fatherly expressions of love, acceptance, understanding, validation. (This is the “raw data”; matching features in current situations are triggers of the whole schema.) A mental model or set of linked, learned constructs operating as living knowledge of a problem and a solution: The problem: knowledge of a vulnerability to a specific suffering. Confident assertiveness in any degree inflicts crushing oppression on others and is hated by them. I would be horrible like Dad and hated by others, as he is, if I asserted my own knowledge or wishes confidently. (This is a model of how the world is, and current situations that appear relevant to this model are triggers of the whole schema.) The solution: knowledge of an urgent broad strategy and concrete tactic(s) for avoiding that suffering. Never express any confident assertiveness, to avoid being horrible and hated (general strategy and pro-symptom purpose), by vigilantly noticing any definite knowledge or opinions forming in myself and blocking them from expression by generating potently self-doubting, self-invalidating thoughts (concrete tactic and manifested symptom).

Emotional schemas can be brought to light during a variety of ways, including Focusing, IFS, and imagining yourself doing something and seeing what you expect to happen as a result.

But suppose that you do manage to bring up a schema which seems wrong to you. What do you do then?

Memory reconsolidation: updating the emotional learning

The formation of memory traces involves consolidation, when the memory is first laid out in the brain; deconsolidation, when an established memory is “opened” and becomes available for changes; and reconsolidation, when a deconsolidated memory (along with possible changes) is stored and becomes frozen again. The term “reconsolidation” is also used to refer to the general process from deconsolidation to reconsolidation; UtEB generally applies the term to mean the entire process. Unless the context indicates otherwise, I do the same.

UtEB reviews some of the history of memory research. Until 1997, neuroscientists believed that past emotional learning became permanently locked in the brain, so that memories could only consolidate, never de- or reconsolidate. More recent research has indicated that once a memory becomes activated, it is temporarily unlocked, allowing it to be changed or erased.

Starting from 2004, new studies suggested that activation alone is not sufficient to deconsolidate the memory. The memories are used to predict that things will occur in a similar fashion as they did previously. Besides just activation, there has to be a significant mismatch between what one experiences and what the memory suggests is about to happen. The violation of expectation can be qualitative (the predicted outcome not occurring at all) or quantitative (the magnitude of the outcome not being fully predicted). In either case, it is this prediction error which triggers the deconsolidation and subsequent reconsolidation.

The memory erasure seems to be specific to the interpretation from which the prediction was produced. For example, someone who has had an experience of being disliked may later experience being liked. This may erase the emotional generalization “I am inherently dislikable”, but it will not erase the memory of the person also having been disliked.

Applied reconsolidation: an example of the schema update process

So, assuming that the model outlined above is correct, how does one apply it in practice?

From what we have discussed so far, the essential steps of erasing a learned belief (including an emotional schema) involves identifying it, activating it, and then finding a mismatch between its prediction and reality.

The first difficulty is that the beliefs involved with the schema are not necessarily consciously available at first. Richard knew that he suffered from a lack of self-esteem, but he was not aware of its reason. The process started from him describing in concrete details how this manifested: as skeptical self-talk during a daily meeting.

As he was guided to imagine what would happen if he didn’t have those thoughts and acted confidently, his therapist was seeking to retrieve the implicit schema and bring it into consciousness so that its contents would become available for access. Once it had been retrieved, the therapist and Richard worked together to express the belief in the schema in maximally emotional language:

“Feeling any confidence means I’m arrogant, self-centered, and totally insensitive like Dad, and people will hate me for it, so I’ve got to never feel confident, ever.”

The authors have developed a therapeutic approach called Coherence Therapy, whose steps closely follow the steps of the memory reconsolidation process. The example of Richard is from this school of therapy.

In Coherence Therapy (as well as related approaches, such as Internal Family Systems), one initially avoids any impulse to argue or disprove with the retrieved schema. This would risk it being pushed away before it has become sufficiently activated to allow for reconsolidation.

Instead, one stays with it. Richard was given a card with the above phrase and instructed to review it every day until the next therapy session, just feeling the extent to which it felt true to him. This served to further integrate access to the schema in question, making it better consciously available.

Because Richard had made the implicit assumptions in his schema explicit, he was able to consciously notice a situation which seemed to violate those assumptions: a prediction mismatch. His therapist recognized this as a piece of contradictory knowledge which could be used to update the old schema. The therapist then guided Richard through a process intended to activate the old schema while bringing the contradictory information into awareness, triggering a reconsolidation process.

The therapist first instructed Richard to mentally bring himself back to the situation where he had just thought of the solution, but held it back. To properly activate the schema, the therapist guided Richard’s attention to the purpose behind his reluctance and Richard’s certainty of any confidence making him disliked. Next, the therapist told Richard to re-live what happened next: the other person making the same suggestion, and the other people in the room looking pleased rather than angry.

The book then has a transcript of the therapist guiding Richard to repeat this juxtaposition of the old schema and the disconfirming experience (italicized brackets in the original):

Afterwards, the therapist and Richard wrote a new card together, which Richard was told to review daily:

All along it’s been so clear that if I confidently say what I know, I will always come across as arrogant, insensitive, and dominating like Dad, and be hated for it. And it’s so weird, looking around the room and seeing that it doesn’t come across like that.

The purpose of the card was to provide additional juxtaposition experiences between the old schema and the new knowledge. While the original transformation sequence might have been enough to eliminate the old schema, the schema might also have been stored in the context of many different situations and contained in several memory systems. In such a situation, further juxtapositions would have helped deal with it.

In a follow-up meeting, Richard reported having lost the feelings of self-doubt, and that speaking up no longer felt like it was any big deal. To verify that the old schema really had lost its power, the therapist tried deliberately provoking his old fears again:

Dropping his voice to a quieter tone, the therapist added, “But tell me, when you have something to say and just say it, what about the danger of coming across as a know-it-all, like Dad, and being hated for that? What about your fear of that and how urgent it is to protect yourself from that?” [...]Richard took in the question, gazed at the therapist in silence for a few seconds, and then replied, “Well, I don’t know what to tell you. All I can say is, that doesn’t trouble me any more. And hearing you say it, it seems a little strange that it ever did—like, what was my problem?”Applied reconsolidation: the schema update process in general

Now that we have looked at a specific example, we can look at a more general version of the process.

Accessing sequence

In Coherence Therapy, the accessing sequence is the preliminary phase of making both a person’s implicit schema and some disconfirming knowledge accessible, so that they can be used in the juxtaposition process:

1. Symptom identification. Establishing which specific symptoms the person regards as problematic, and when and where they manifest. In Richard’s case, the general symptom was a lack of confidence, which specifically manifested as negative self-talk in meetings.
2. Retrieval of target learning. Bringing into explicit awareness the purpose behind the symptoms. This can then be used to guide the search for disconfirming knowledge, as well as accessing the original schema in order to reconsolidate it. In Richard’s case, the purpose was to avoid expressing confidence in a way that would make people hate him.
3. Identification of disconfirming knowledge. Identifying some past or present experience which directly contradicts the original learning. This knowledge does not necessarily need to feel “better” or “more positive” than the old one, just as long as it is mutually exclusive with the old one. In Richard’s case, the disconfirming knowledge was the experience of his co-worker confidently proposing a solution and being well-received.
Erasure sequence

Once both the target schema and the disconfirming knowledge are known, the erasure steps can be applied to update the learning:

1. Reactivation of the target schema. Tapping into the felt truth of the original learning, experiencing it as vividly as possible.
2. Activation of disconfirming knowledge, mismatching the target schema. Activating, at the same time, the contradictory belief and having the experience of simultaneously believing in two different things which cannot both be true.
3. Repetitions of the target-disconfirmation pairing.

Something that the authors emphasize is that when the target schema is activated, there should be no attempt to explicitly argue against it or disprove it, as this risks pushing it down. Rather, the belief update happens when one experiences their old schema as vividly true, while also experiencing an entirely opposite belief as vividly true. It is the juxtaposition of believing X and not-X at the same time, which triggers an inbuilt contradiction-detection mechanism in the brain and forces a restructuring of one’s belief system to eliminate the inconsistency.

The book notes that this distinguishes Coherence Therapy from approaches such as Cognitive Behavioral Therapy, which is premised on treating some beliefs as intrinsically irrational and then seeking to disprove them. While UtEB does not go further into the comparison, I note that this is a common complaint that I have heard of CBT: that by defaulting to negative emotions being caused by belief distortions, CBT risks belittling those negative emotions which are actually produced by correct evaluations of the world.

I would personally add that not only does treating all of your beliefs - including emotional ones - as provisionally valid seem to be a requirement for actually updating them, this approach is also good rationality practice. After all, you can only seek evidence to test a theory, not confirm it.

If you notice different parts of your mind having conflicting models of how the world works, the correct epistemic stance should be that you are trying to figure out which one is true - not privileging one of them as “more rational” and trying to disprove the other. Otherwise it will be unavoidable that your preconception will cause you to dismiss as false beliefs which are actually true. (Of course, you can still reasonably anticipate the belief update going a particular way - but you need to take seriously at least the possibility that you will be shown wrong.)

Similar to what Eliezer previously suggested, this can actually be a relief. Not only would it be an error as a matter of probability theory to try to stack the deck towards receiving favorable evidence, doing so would also sabotage the brain’s belief update process. So you might as well give up trying to do so, relax, and just let the evidence come in.

I speculate that this limitation might also be in place in part to help avoid the error where you decide which one of two models is more correct, and then discard the other model entirely. Simultaneously running two contradictory schemas at the same time achieves good communication within the brain, as it allows both of them to be properly evaluated and merged rather than one of them being thrown away outright. I suspect that in Richard’s case, the resulting process didn’t cause him to entirely discard the notion that some behaviors will make him hated like his dad was - it just removed the overgeneralization which had been produced by having too little training data as the basis of the schema.

Of course, this means that there does need to be some contradictory information available which could be used to disprove the original schema. One might have a schema for which no disconfirmation is available because it is correct, or a schema which might or might not be correct but which is making things worse and cannot easily be disconfirmed. UtEB mentions the example of a man, "Tómas", who had a desire to be understood and validated by someone important in his life. Tómas remarked that a professional therapist who was being paid for his empathy could never fulfill that role. The update contradicting the schema that nobody in his life really understood him, would have to come from someone actually in his life.

Another issue that may pop up with the erasure sequence is that there is another schema which predicts that, for whatever reason, running this transformation may produce adverse effects. In that case, one needs to address the objecting schema first, essentially be carrying out the entire process on it before returning to the original steps. (This is similar to the phenomenon in e.g. Internal Family Systems, where objecting parts may show up and have their concerns addressed before work on the original part can proceed.)

Verification step

Finally, after the erasure sequence has been run, one seeks to verify that lasting change has indeed happened and that the target schema has been transformed. UtEB offers the following behavioral markers as signs that a learning which has previously generated emotional responses has in fact been erased:

• “A specific emotional reaction abruptly can no longer by reactivated by cues and triggers that formerly did so or by other stressful situations.”
• “Symptoms of behavior, emotion, somatics, or thought that were expressions of that emotional reaction also disappear permanently.”
• “Non-recurrence of the emotional reaction and symptoms continues effortlessly and without counteractive or preventive measures of any kind.”

The authors interpret current neuroscience to say that only memory reconsolidation can produce these kinds of markers. They cannot be produced by counteractive or competitive processes, such as trying to learn an opposite habit to replace a neurotic behavior. Counteractive processes are generally fragile and susceptible to relapse. When these markers are observed in clinical work, UtEB argues that one may infer that reconsolidation has led to the original learning being replaced.

Further examples

For additional examples of the schema update process, I recommend reading the book, which contains several more case studies of issues which were dealt with using this approach. Here’s a brief summary of the most detailed ones (note that some of these examples are actually more detailed and include additional complications, such as more than one symptom-producing schema; I have only summarized the most prominent ones to give a taste of them):

• “Charlotte”. Issue: obsessive attachment to a former lover. Schema: “It would be much better if I was merged with my lover”. Contradictory knowledge: The harm caused by not having boundaries.
• “Ted”. Issue: an inability to hold a steady job and a general lack of success in life. Schema: “If my life is a mess, my father will be forced to admit how badly he screwed up as a parent.” Contradictory knowledge: Realizing that Ted’s father would never admit failure, no matter what.
• “Brenda”. Issue: stage fright when having a leading role in an upcoming play. Schema: being on the stage in front of an audience means being unable to get off, causing helplessness similar to when Brenda was in a car with her alcoholic father and couldn’t get off. Contradictory knowledge: re-imagining the scene and the way how Brenda could actually have gotten out of the car.
• “Travis”. Issue: inability to experience intimate emotional closeness in relationships. Schema: “Nobody will pay attention to how I feel or give me understanding for how I’m hurting. I don’t matter, and I’m all on my own.” Contradictory knowledge: the therapist’s empathic presence and listening.
• “Regina”. Issue: strong anxiety and panic during/after interacting with other people. Schema: “I’m acceptable and lovable only if I do everything perfectly.” Contradictory knowledge: Regina’s Uncle Theo loves her regardless of what imperfections she might have.
• “Carol”. Issue: wanting to avoid sex with her husband despite feeling emotionally close to him. Schema: engaging in any sexuality means being overtly sexual and harming Carol’s daughter, in the way that Carol was harmed by her mother’s overt sexuality. Contradictory knowledge: once the schema was made conscious, it activated the brain’s spontaneous mismatch detection mechanisms and started to feel silly.

As the last item suggests, sometimes just making a schema explicit is enough to start to dismantle it. The authors suggest that the brain has a built-in detection system which compares any consciously experienced beliefs for inconsistencies with other things that a person knows, and can spontaneously create juxtaposition experiences by bringing up such inconsistent information. They suggest that therapies which are based on digging up previously unconscious material, but which do not have an explicit juxtaposition step, work to the extent that the uncovered material happens to trigger this spontaneous mismatch detection. (We already saw this happening with Richard - once his underlying schema had been made conscious, he was startled to later notice what seemed like a contradiction.)

One may note the connection to the model in Consciousness and the Brain that when some subsystem in the brain manages to elevate a mental object into the content of consciousness, multiple subsystems will synchronize their processing around that object. If the object is an explicit belief, then any subsystem which is paying attention to that object may presumably detect inconsistencies with that subsystem’s own models.

Besides these case studies from Coherence Therapy, the authors also analyze published case studies from Accelerated Experiental Dynamic Psychotherapy, Emotion-Focused Therapy, Eye-Movement Desensitization and Reprocessing, and Interpersonal Neurobiology. They try to show how these cases also carried out a juxtaposition process, even if the theoretical frameworks of those therapies did not explicitly realize it. It is the claim of the authors that any therapy which causes lasting emotional change does it through reconsolidation. Finally, the book contains four essays from other therapists (using Coherence Therapy and EMDR), who analyze some of their own case studies.

Evaluating the book’s plausibility

Now that we have looked at the book’s claims, let’s look at whether we should believe in them.

It is unclear to me how reliable the neuroscience results are; the authors cite a number of studies, but each individual claim only references a relatively small number of them.

On a brief look, I could not find any reviews or papers that would have directly made a critical assessment of the book’s model. However, I found something that might be even better.

Behavioral and Brain Sciences is a respected journal covering subject areas across the cognitive sciences. BBS publishes “target articles” which present some kind of a thesis or review about a particular topic, together with tens of brief commentaries which respond to the target article, and a final response by the target article’s authors to the commentaries.

In 2015, four prestigious (with a total of 500 published research articles between them) psychologists published a BBS target article, Memory reconsolidation, emotional arousal, and the process of change in psychotherapy: New insights from brain science (Lane et al. 2015). While the exact model that they outline has a number of differences from the UtEB model, the core idea is the same: that therapeutic change from a wide variety of therapeutic approaches, “including behavioral therapy, cognitive-behavioral therapy, emotion-focused therapy, and psychodynamic psychotherapy, results from the updating of prior emotional memories through a process of reconsolidation that incorporates new emotional experiences.”

One interesting difference was that Lane et al. describe emotional schemas somewhat differently. In their model, the schemas form memory structures with three mutually integrated components: emotional responses, episodic/autobiographical memories, and semantic structures (e.g. abstract beliefs which generalize over the various incidents, such as the claim that “people are untrustworthy”). Any of these components can be used as an entry point to the memory structure, and can potentially update the other components through reconsolidation. They hypothesize that different forms of therapy work by accessing different types of components: e.g. behavior therapy and emotion-focused therapy access emotional responses, conventional psychoanalysis uses access to biographical memories, and cognitive behavioral therapy accesses semantic structures.

I read the target article, all the commentaries, and the responses. Given the similarities between Lane et al.’s model and the UtEB model, I think we can consider the responses to Lane et al. to generally offer a useful evaluation of the UtEB model as well.

One significant difference which needs to be noted is that Lane et al.’s model of memory reconsolidation does not mention the requirement for a prediction mismatch before reconsolidation can happen. This was remarked on in the response from UtEB’s authors. In their counter-response, Lane et al. noted UtEB’s model to be highly compatible with theirs, and remarked that further research is needed to nail down the conditions which make reconsolidation the most effective.

The other responses to Lane et al. were mostly from psychologists, psychiatrists, and neuroscientists, but also included the occasional economist, philosopher, philologist and folklorist. Several of the responses were generally positive and mostly wanted to contribute additional details or point out future research directions.

However, there were also a number of skeptical responses. A common theme which emerged from several concerned the limitations of the current neuroscience research on memory reconsolidation. In particular, most of the studies so far have been carried out on rats, and specifically testing the elimination of a fear response to electric shocks. As one of the responses points out, “neither the stimuli nor the subjects are generalizable to the kind of rich autobiographical memories involved in therapy.” A number also raised the question whether all therapeutic change really involves reconsolidation, as opposed to some related mechanism, such as creating new memory structures which compete with the original as opposed to replacing it.

My non-expert reading is that the critical responses are right in that a gap remains between the clinical and behavioral findings on the other hand, and the neuroscience findings on the other. There are various patterns which can be derived from psychological research and clinical therapy experience, and a small number of neuroscience findings which establish the existence of something that could explain those patterns. However, the neuroscience findings have only been established in a rather narrow and limited context; the connection between them and the higher-level patterns is a plausible link, but it remains speculative nonetheless.

Personally I consider the book’s model tentatively promising, because it seems to explain many observations which I had independently arrived at before reading it. For example, I had noticed an interesting thing with anxieties, where I let e.g. a sensation of social anxiety stay active in my mind, neither accepting it as truth nor pushing it away while I went to do social things. This would then cause the anxiety to update, making me feel less anxious if it was indeed the case that the social interaction was harmless. This fits nicely together with the framework of an activated memory structure becoming open to reconsolidation and then being updated by a prediction mismatch (the situation not being as bad as expected).

Likewise, in my post Integrating disagreeing subagents, I reviewed a variety of rationality and therapeutic techniques, and suggested that they mostly worked either by merging or combining two existing models that a person’s brain already had, or augmenting the existing models by collecting additional information.

In particular, I considered an example from cognitive behavioral therapy, where a man named Walter was feeling like he was impossible to be in a relationship with after he had broken up with his boyfriend. At the same time, he did not think that someone else breaking up with their partner was an indication of them being impossible to be in a relationship with. He and his therapist role-played an interaction where the therapist pretended to be a friend who had recently broken up, and Walter explained why this did not make the friend a relationship failure. In the process of doing so, Walter suddenly realized that he wasn’t a failure, either.

I commented:

Walter was asked whether he’d say something harsh to a friend, and he said no, but that alone wasn’t enough to improve his condition. What did help was putting him in a position where he had to really think through the arguments for why this is irrational in order to convince his friend, and then, after having formulated the arguments once himself, get convinced by them himself.In terms of our framework, we might say that a part of Walter’s mind contained a model which output a harsh judgment of himself, while another part contained a model which would output a much less harsher judgment of someone else who was in otherwise identical circumstances. Just bringing up the existence of this contradiction wasn’t enough to change it: it caused the contradiction to be noticed, but didn’t activate the relevant models extensively enough for their contents to be reprocessed.But when Walter had to role-play a situation where he thought of himself as actually talking with a depressed friend, that required him to more fully activate the non-judgmental model and apply it to the relevant situation. This caused him to blend with the model, taking its perspective as the truth. When that perspective was then propagated to the self-critical model, the easiest way for the mind to resolve the conflict was simply to alter the model producing the self-critical thoughts.

This seems like a straightforward instance of belief juxtaposition, and one where I ended up independently deriving something like UtEB’s memory reconsolidation model: I too noted that the relevant belief structures need to be simultaneously activated in the right way to allow for the brain to revise one of them after noticing the contradiction. In general, UtEB’s model of how things work rings true in my experience, making me inclined to believe that its description of how therapy works is correct, and that its model of how it is connected to neuroscience might also be.

UtEB and the subagent model

As many readers know, I have been writing a sequence of posts on multi-agent models of mind. In Building up to an Internal Family Systems model, I suggested that the human mind might contain something like subagents which try to ensure that past catastrophes are not repeateddo not repeat. In subagents, coherence, and akrasia in humans, I suggested that behaviors such as procrastination, indecision, and seemingly inconsistent behavior result from different subagents having disagreements over what to do.

As I already mentioned, my post on integrating disagreeing subagents took the model in the direction of interpreting disagreeing subagents as conflicting beliefs or models within a person’s brain. Subagents, trauma and rationality further suggested that the appearance of drastically different personalities within a single person might result from unintegrated memory networks, which resist integration due to various traumatic experiences.

This post has discussed UtEB’s model of conflicting emotional schemas in a way which further equates “subagents” with beliefs - in this case, the various schemas seem closely related to what e.g. Internal Family Systems calls “parts”. In many situations, it is probably fair to say that this is what subagents are.

That said, I think that while this covers a very important subset of subagents, not everything which I have been referring to as a subagent falls straightforwardly under the belief-schema model. In subagents and neural Turing machines as well as Against “System 1” and “System 2”, I also covered subagents in a more general way, as also including e.g. the kinds of subsystems which carry out object recognition and are used to carry out tasks like arithmetic. This was also the lens through which I looked at subagents in my summary of Consciousness and the Brain. Which kind of view is the most useful, depends on exactly what phenomenon we are trying to understand.

Discuss

### Characterizing Real-World Agents as a Research Meta-Strategy

8 октября, 2019 - 18:32
Published on October 8, 2019 3:32 PM UTC

Background

Intuitively, the real world seems to contain agenty systems (e.g. humans), non-agenty systems (e.g. rocks), and ambiguous cases which display some agent-like behavior sometimes (bacteria, neural nets, financial markets, thermostats, etc). There’s a vague idea that agenty systems pursue consistent goals in a wide variety of environments, and that various characteristics are necessary for this flexible goal-oriented behavior.

But once we get into the nitty-gritty, it turns out we don’t really have a full mathematical formalization of these intuitions. We lack a characterization of agents.

To date, the closest we’ve come to characterizing agents in general are the coherence theorems underlying Bayesian inference and utility maximization. A wide variety of theorems with a wide variety of different assumptions all point towards agents which perform Bayesian inference and choose their actions to maximize expected utility. In this framework, an agent is characterized by two pieces:

• A probabilistic world-model
• A utility function

The Bayesian utility characterization of agency neatly captures many of our intuitions of agency: the importance of accurate beliefs about the environment, the difference between things which do and don’t consistently pursue a goal (or approximately pursue a goal, or sometimes pursue a goal…), the importance of updating on new information, etc.

Sadly, for purposes of AGI alignment, the standard Bayesian utility characterization is incomplete at best. Some example issues include:

• The need for a cartesian boundary - a clear separation between “agent” and “environment”, with well-defined input/output channels between the two
• Logical omniscience - the assumption that agents can fully compute all of the implications of the information available to them, and track every possible state of the world
• Path independence and complete preferences - the assumption that an agent doesn’t have a general tendency to stay in the state it’s in

One way to view agent foundations research is that it seeks a characterization of agents which resolves problems like the first two above. We want the same sort of benefits offered by the Bayesian utility characterization, but in a wider and more realistic range of agenty systems.

Characterizing Real-World Agents

We want to characterize agency. We have a bunch of real-world systems which display agency to varying degrees. One obvious strategy is to go study and characterize those real-world agenty systems.

Concretely, what would this look like?

Well, let’s set aside the shortcomings of the standard Bayesian utility characterization for a moment, and imagine applying it to a real-world system - a financial market, for instance. We have various coherence theorems saying that agenty systems must implement Bayesian utility maximization, or else allow arbitrage. We have a strong prior that financial markets don’t allow arbitrage (except perhaps very small arbitrage on very short timescales). So, financial markets should have a Bayesian utility function, right? Obvious next step: pick an actual market and try to figure out its world-model and utility function.

I tried this, and it didn’t work. Turns out markets don’t have a utility function, in general (in this context, it’s called a “representative agent”).

Ok, but markets are still inexploitable and still seem agenty, so where did it go wrong? Can we generalize Bayesian utility to characterize systems which are agenty like markets? This was the line of inquiry which led to “Why Subagents?”. The upshot: for systems with internal state (including markets), the standard utility maximization characterization generalizes to a multi-agent committee characterization.

This is an example of a general strategy:

• Start with some characterization of agency - don’t worry if it’s not perfect yet
• Apply it to a real-world agenty system - specifically, try to back out the characterizing properties, e.g. the probabilistic world-model and utility function in the case of a Bayesian utility characterization
• If successful, great! We’ve gained a useful theoretical tool for an interesting real-world system.
• If unsuccessful, first check whether the failure corresponds to a situation where the system actually doesn’t act very agenty - if so, then that actually supports our characterization of agency, and again tells us something interesting about a real-world system.
• Otherwise, we’ve found a real-world case where our characterization of agency fails. Look at the system’s actual internal behavior to see where it differs from the assumptions of our characterization, and then generalize the characterization to handle this kind of system.

Note that the last step, generalizing the characterization, still needs to maintain the structure of a characterization of agency. For example, prospect theory does a fine job predicting the choices of humans, but it isn’t a general characterization of effective goal-seeking behavior. There’s no reason to expect prospect-theory-like behavior to be universal for effective goal-seeking systems. The coherence theorems of Bayesian utility, on the other hand, provide fairly general conditions under which Bayesian induction and expected utility maximization are an optimal goal-seeking strategy - and therefore “universal”, at least within the conditions assumed. Although the Bayesian utility framework is incomplete at best, that’s still the kind of thing we’re looking for: a characterization which should apply to all effective goal-seeking systems.

Some examples of (hypothetical) projects which follow this general strategy:

• Look up the kinetic equations governing chemotaxis in e-coli. Either extract an approximate probabilistic world-model and utility function from the equations, find a suboptimality in the bacteria’s behavior, or identify a loophole and expand the characterization of agency.
• Pick a financial market. Using whatever data you can obtain, either extract (not necessarily unique) utility functions and world models of the component agents, find an arbitrage opportunity, or identify a new loophole and expand the characterization of agency.
• Start with the weights from a neural network trained on a task in the openai gym. Either extract a probabilistic world model and utility function from the weights, find a strategy which dominates the NN’s strategy, or identify a loophole and expand the characterization of agency

… and so forth.

Why Would We Want to Do This?

Characterization of real-world agenty systems has a lot of advantages as a general research strategy.

First and foremost: when working on mathematical theory, it’s easy to get lost in abstraction and lose contact with the real world. One can end up pushing symbols around ad nauseum, without any idea which way is forward. The easiest counter to this failure mode is to stay grounded in real-world applications. Just as a rationalist lets reality guide beliefs, a theorist lets the problems, properties and intuitions of the real world guide the theory.

Second, when attempting to characterize real-world agenty systems, one is very likely to make some kind of forward progress. If the characterization works, then we’ve learned something useful about an interesting real-world system. If it fails, then we’ve identified a hole in our characterization of agency - and we have an example on hand to guide the construction of a new characterization.

Third, characterization of real-world agenty systems is directly relevant to alignment: the alignment problem itself basically amounts to characterizing the wants and ontologies of humans. This isn’t the only problem relevant to FAI - tiling and stability and subagent alignment and the like are separate - but it is basically the whole “alignment with humans” part. Characterizing e.g. the wants and ontology of an e-coli seems like a natural stepping-stone.

One could object that real-world agenty systems lack some properties which are crucial to the design of aligned AGI - most notably reflection and planned self-modification. A theory developed by looking only at real-world agents will therefore likely be incomplete. On the other hand, you don’t figure out general relativity without figuring out Newtonian gravitation first. Our understanding of agency is currently so woefully poor that we don’t even understand real-world systems, so we might as well start with that and reap all the advantages listed above. Once that’s figured out, we should expect it to pave the way to the final theory: just as general relativity has to reproduce Newtonian gravity in the limit of low speed and low energy, more advanced characterizations of agency should reproduce more basic characterizations under the appropriate conditions. The subagents characterization, for example, reproduces the utility characterization in cases where the agenty system has no internal state. It all adds up to normality - new theories must be consistent with the old, at least to the extent that the old theories work.

Finally, a note on relative advantage. As a strategy, characterizing real-world agenty systems leans heavily on domain knowledge in areas like biology, machine learning, economics, and neuroscience/psychology, along with the math involved in any agency research. That’s a pretty large pareto skill frontier, and I’d bet that it’s pretty underexplored. That means there’s a lot of opportunity for new, large contributions to the theory, if you have the domain knowledge or are willing to put in the effort to acquire it.

Discuss

### What's the dream for giving natural language commands to AI?

8 октября, 2019 - 16:42
Published on October 8, 2019 1:42 PM UTC

0

I promised in a previous post that I would give a post-mortem for a scheme for learning the intentional stance from natural language. This is that post. But first, I should explain why such an idea might seem good in the first place.

Some people think of AI as a genie. The goal of AI research, in this picture, is to "tell the AI what to do," sometimes explicitly in natural language. And then since the AI is smart, it will understand what we mean and do that, because to do something else would be stupid.

This is, in a sense, very naive. Making an AI that does what we want is not at all like instructing a human - see the relevant Eliezer post - the methods, dangers, and goals are all different. But... if the AI understood what we meant, maybe we could just tell it what to do.

Of course, "understood what we meant" captures more or less the whole problem, because meaning isn't like the charge of the electron, it's nowhere in the words themselves. When you understand moral language, you're implicitly using your morals. But what if we trained an AI so that it functionally understood moral language - would that be implicitly using your morals too, and isn't that exactly what we want?

1 - The Artificial Intentional Stance

I like to think of myself as having preferences, but at the same time I am made of atoms, and my preferences are not-like-the-charge-of-the-electron, they're nowhere in the atoms. Instead, my preferences are an abstraction that I (and others) use when thinking about me.

So part of the this artificial intentional stance stuff can be summed up as: get the AI to think about humans like humans think about humans. (Another part is that abstractions are contagious. If I want to go to the gym, to handle this correctly this you need abstractions not just for me but also for the gym.)

We often put too much magic into the word "understand." If the AI can hold a good conversation and extract real-world information from human speech, it's reasonable to say it understands what we're saying. And then once it understands us, you might think "communicating our goals to it is a lot like communicating with a human."

But it's easy to hold a decent conversation without taking the intentional stance towards humans, and easier still to extract real-world information from human speech without the intentional stance. This leads to problems that become clear if you try to take an AI that does a good job at modeling language, and follow step by step how to get it to choose actions that are good for humans.

The dream is to learn the intentional stance by using the information implicit in our use of language. The intentional stance requires picking out good levels of abstraction to model humans on, and using language implies that the good levels of abstraction are the ones humans implicitly use in language. Is this what we want? I don't know, it might be?

It certainly isn't the only option - we might imagine other schemes involving trying to amplify emulations of humans, semi-supervised learning from examples of good and bad behavior, or multi-stage chains of making increasingly trustworthy AIs. But the question is whether it's an option.

3 - Process

It's not hard to hook up a videocamera to an image captioner to a deep reinforcement learner and say you can input goals with natural language because when you set the goal to "cat," your camera will look for cats. It's a lot harder to get that camera to look for what's best in life.

This is the bind I got myself into, writing this post. Value learning schemes that are simple are wrong (issues with the cat camera above are left as an exercise to the reader), and value learning schemes that seem promising have been selected for incomprehensibility and poor epistemic luck. So I tried to split the difference, if favoring interestingness over simplicity.

Here's some of the rules of thumb I used when thinking of ways to apply natural language processing to value learning:

First, I wanted to avoid the scheme having glaringly unspecified parts. It's very easy to be lazy and not specify something enough that it actually chooses actions, or feel like I've made progress but not be able to apply it. Usually either of these meant I was sweeping problems under the rug - the right level of specificity involves sweeping out some of those cobwebs.

Second, I needed to encourage myself to be specific about the intended purpose of natural language processing in each particular scheme. Yes, the dream is that it "includes common sense" or something like that, but that's not specific enough mental technology to tell whether you're solving the intended problem without unnecessary side effects, or explain why different methods get different results.

It was profitable to think of natural language processing as being targeted at the problem of alien concepts: when the AI can match your training examples but still fail to generalize how you want because it's representing your examples in an alien way. For example, an image classifier might learn to distinguish dogs by the texture of their fur, but we're not going to be happy with how that generalizes to fur-less dogs or dog-less fur. Now replace "dogs" with "human values" and "fur" with "superficial features that work well on the training set."

An even more specific purpose of natural language would be "greedy reification" - actively trying to form concepts that correspond to linguistic tokens. So if we have a word "dog," we want to incentivize the AI to form a concept that picks out dogs in the world-model, and then the hope is that this also works on "human values."

4 - Mort

So here's a value learning scheme: try to squish the world and natural language into the same latent space, just with different input/output functions.

Training this simultaneous model might just be separately trying to do encoding-decoding or prediction tasks with sensory data and text, but more plausibly it should involve translation tasks where we can associate words with sensory environments. The model required is somewhat subtle, because we don't want words associated with raw sense data, we want words associated with the state of the AI's model of the world. This mandates that to the world-model, this latent space should look like the persistent state associated with sequence prediction or encoding-decoding of sequences of sense data, with transition dynamics partially included in the shared information. This means the language model should also look like sequence prediction or encoding with some local state consisting of the high-level features.

If I haven't said anything impossible so far, we could use sufficiently advanced technology to train this simultaneous model so that it's good at understanding both the world and language, and competent at turning one into the other when it comes to simple training examples. Can you now solve value learning by giving it a bunch of English descriptions what we want ("human values satisfied," "do the right thing," "a fulfilling and cosmopolitan future for the galaxy," et c.), and coding it to choose actions that make the state of the world like that?

Looking on the bright side first, what advantages might this have?

• It leverages the structure of language in a way that might be what we want, by training its shared high-level features to be suitable for language modeling as well as world-modeling. This is that greedy reification property - the label "dog" gets a representation that captures the functional properties of the "dog" label in human speech and description of the world, which is at least a higher order of problem than the image classifier that identifies "dog" with the texture of dog fur.
• The knowledge of the world also impacts the verbal reasoning. If we ask it to do some difficult verbal task involving rotating imagined objects, the dream is that it could take advantage of some shared representation that would code for rotating objects on the world-model side, and so has the right dynamics.
• Scaling seems to work in our favor. To the extent that this model follows our naive notion of "understanding," the better it understands what we mean the better job it will do.
• With small modifications, it could mimic CIRL's state of uncertainty about human values. It would need to keep track of (and properly update) uncertainty in its model and propagate that forward to uncertainty in its reward, which it can then maximize by picking strategies that account for value of information.

What are some big issues with this? Take a second if you like.

No, really, I'd be interested in what people come up with on their own. I don't understand this family of schemes as well as I'd like.

• 1) The communication between the two sides could be incomplete. In fact, we would expect there to be parts of the shared latent space that one side ignores but get treated as useful information by the other, if they get separate training. This is worrisome if we want to rely on translating a difficult verbal concept into an opinion on states of the world - what if key information is encoded in the features that the other encoder has learned to ignore? To the extent that the phrase "human values" does help us predict the world, it seems like this should be one of the problems that decreases with increasing competence, but I'm not totally sure.
• 2) This simple scheme has zero extra protection from unintended maxima. Goodhart's law rears its ugly head - if our target is a particular state of the model of the world, then maybe there are nearby world-states that score well on the objective function but are "unnatural" in some way. This is a general problem that appears any time you have a system trained on prediction but asked to do intervention. How does this square with using natural language to learn non-alien concepts? It seems plausible that natural language might help concepts generalize to new contexts that are well-suited to language, but still leave some unnatural contexts lurking around misclassified.
• 3) At first blush, this doesn't care about metaethics. It doesn't take into account what we'd like to like, or how we think about values, in order to change its own function. It's not a process that learns about humans and then outputs a decision procedure, it's a fixed decision procedure that extensively uses learning about humans. It's possible that this could actually be patched by putting our meta-ethical desires into the verbal "wish" - or even our desire for it to learn a model of meta-ethics. Each such kludge seems to increase the difficulty of the task, but I'm not sure if anything actually breaks (or if anything actually gets solved).
• 4) (3) wouldn't be a problem if this were meta-ethically great, but it's probably not. The most glaring example is that it aims for a fixed goal state at an approximately fixed level of abstraction, which may not be what we'd want on reflection. If we think of the goal state as living in some "slice" along the translation of features from language to microphysical states of the world, it's conceivable that we choose (or set some mysterious hand-tuned parameter) to put this slice at the wrong place (or maybe there is no right place). Using planning towards a goal also means that it's happy to violate deontological rules, and take you apart if it can sue your atoms for something better.

5 - Wrap-up

Going back to the artificial intentional stance and the problem of alien concepts, it seems like this helps in some ways but not in others.

It seems to help with the intentional stance at the object level - the everyday work of translating "Charlie wants to go to the gym" into reasonable actions - but not at the meta-level. It's doubtful that it's modeling humans how they want to be modeled. Maybe this indicates that it would be profitable to break down this concept further. It also might spark your imagination about how to take the same information about humans and end up with something that models humans in a variable way.

A different thing is going on in the department of alien concepts, where we've run into the stress that Goodhart's law places on concepts. Instead of thinking about human-modeling, this makes me want to focus on the decision procedure and training. Can we find a decision procedure that leverages prediction and translation tasks in a way that puts less stress on the concepts? Can we use a training procedure that reduces the context shift when trying to use the model to choose actions?

Overall I think this avenue is pretty interesting to think about. Maybe this also serves as a concrete example of what I mean by trying to create the artificial intentional stance, which can be generalized from language to other options for learning about humans.

Discuss

### Artifact: What Went Wrong?

8 октября, 2019 - 15:10
Published on October 8, 2019 12:10 PM UTC

Artifact had every advantage. Artifact should have been great. Artifact was great for the right players, and had generally positive reviews. Then Artifact fell flat, its players bled out, its cards lost most of their value, and the game died.

Valve takes the long view, so the game is being retooled and might return. But for now, for all practical purposes, the game is dead.

Richard Garfield and Skaff Elias have one take on this podcast. They follow up with more thoughts in this interview.

Here’s my take, which is that multiple major mistakes were made, all of which mattered, and all of which will be key to avoid if we are to bring the combination of strategic depth and player-friendly economic models back to collectible card games.

I see ten key mistakes, which I will detail below.

Alas, we do not get to run controlled experiments. The lack of ability to experiment and iterate was the meta-level problem with Artifact. The parts of the game that Valve knew how to test, and knew to test, were outstanding, finely crafted and polished. The parts that Valve did not test had severe problems.

We will never know for sure which reasons were most critical, and which ones were minor setbacks. I will make it clear what my guesses are, but they are just that, guesses.

Reason 1: Artifact Was Too Complex, Complicated and Confusing

Artifact streams were difficult to follow even if you knew the game. As teaching tools they didn’t work at all. I heard multiple reports that excellent streamers were unable to explain to viewers what was going on in their Artifact games.

I was mostly able to follow streams when I had a strong strategic understanding of the game and all of its cards and recognized all the cards on sight. Mostly.

I was fortunate to learn Artifact in person with those deeply committed to it, at the Valve offices, and in a setting where I had several days with no distractions to become comfortable with the game and how it worked. All eight of us who were introduced to the game that week got what it was about and had fun. We were able to pass through hours of not knowing why our heroes had died or our cards hadn’t worked the way we expected.

But that’s all an extraordinary bar to entry. It also helped that we were playing against each other, rather than spikes from the internet beating us over the head with netdecks.

Valve did amazingly great work on the user interface that, in addition to making the game beautiful, made it as clear as possible what was going on at all times. It wasn’t enough.

What was enough was the same thing that Magic: The Gathering players need: A human who will sit down with you and explain how things work. No other solution comes close.

Whenever I talked to players who bounced off Artifact, even top professional Magic players, complexity was always the number one complaint.

I introduced Gaudenis Vidugiris to the game. He bounced off. Too complicated.

I introduced Sam Black to the game. He bounced off. Too complicated.

I played in a tournament match during the beta against Andrew Cuneo. He knew the basic rules but had no idea what was going on and felt lost and helpless. He bounced off.

Randy Buehler reported it took him about two weeks to start understanding things well enough to begin having fun.

If those four think your game is too complicated and hard to grasp, I have some news. Your game is too complicated and hard to grasp. There is no large audience that is more sophisticated than that, that wants more complexity than that.

Artifact when you don’t know what is going on is a frustrating experience. I was only able to enjoy playing Artifact when I was playing most days, and making it my strategic gaming focus, and giving games the attention they deserved. When I did that, I had some of the best and most intensive and interesting gaming experiences I can remember. Whenever I tried to phone things in, it became quickly clear there was no point.

Some of us stuck with the game through those problems and got tens or hundreds of hours of good experiences out of the game. But eventually it took its toll on all of us. When Valve started changing cards, I lost heart and gave up, no longer confident that my mastery of this complex information and its strategic implications would last long enough to justify the continued investment.

I loved Artifact. I love complex games and love that they exist and that others love them. It brings me joy to know others are building superstructures in Minecraft, Dwarf Fortress and Europa Universalis. Whenever I was at the World Boardgaming Championships I would devote entire days to games of Advanced Civilization, then I would check on those who devoted their entire week to a single game of Global War, and would have done the same for any games of Empires in Arms. But it is worth noting that as much as I admire them and want to try them, I have never played Global War or Empires in Arms. They take too long, require too much dedication, and I could never make it happen. They’re a bridge or two too far.

The most extreme case might be, interestingly, the game Artifact was based on: Dota 2. Dota 2 takes dozens of hours before one is able to play the game as anything other than a training exercise. I tried to learn it by having someone sit down and play a learning game with me. It lasted an hour. At the end of that hour, I felt only marginally closer to understanding the game. I still was so bad at the game that I suspected I was hurting my team by doing anything at all rather than staying in the starting area doing nothing.

Tutorials continue to be a grand unsolved problem. It’s not clear that most of them aren’t actively counterproductive for games at this level, and we shouldn’t just assign experienced players to teach and mentor newer players. No other tech is known to work (and for Dota 2, even that seemed like it wasn’t great).

Despite these problems, Dota 2 is huge, and hugely successful. Part of that is greater resonance. Part of that is path dependence, as so many players have made the investment and that has justified it to others. Part of that is that the game pays off in hundreds or thousands of hours. But it’s not like I can watch a stream of Dota 2 and have any idea what is going on in the sense that people watching Artifact don’t know what is going on. It’s not like I feel there’s a reasonable path to learning the game, even though I’ve wanted to at various points.

It is easy to see why complexity didn’t seem like a mission critical issue for the Dota 2 card game, when Dota 2 itself arguably has these issues even worse than Artifact. And as Artifcact designers Richard Garfield and Skaff Elias point out, even if you knock out 90%+ of all potential players with this, that still leaves plenty of eager potential participants. The point is a few players who love you, not to appeal to everyone.

I don’t think this had to be fatal on its own, but it put a lot of pressure on the game to deliver the goods elsewhere to make up for it.

Reason 2: Artifact Was Too Skill Testing

Artifact announced it was handing Stanislav Cifka a million dollars.

They worded this as an announcement of a million dollar first prize tournament. Technically, many of us had hope it might be us. But we all knew, in our heart of hearts, it was going to be Stanislav Cifka. He owned all of us throughout the beta, going through entire tournaments without dropping games. The one time I can think of he did lose, it was through a clear tactical error in the final.

I entered one Artifact tournament after the game premiered, and played in several beta tournaments. Results were, shall we say, mixed.

In every case, for every match that I won, I could point to impactful errors by my opponent.

In every case, for every match that I lost, I knew I had made key mistakes and deserved to lose.

Good players in Artifact have a huge advantage against bad players. Great players have the same advantage against good players.

If you had to line up players such that the player in front would dominate the player behind them such that an upset would be very surprising (so let’s say 400 ELO points distance), for Magic: The Gathering you’d get something like (Top Player -> PTQ Level Player -> Reasonable Store Player -> Bad Player) for four players in the row. I can see a case for three or five.

Chess would have (2800 -> 2400 -> 2000 -> 1600 -> 1200 -> 800) so it would have six.

Artifact would have at least six. It might have seven.

That is too much skill testing, especially for a game that people didn’t think was all that skill testing. Players noticed all the bad parts of too much skill testing without the benefits.

What made this hard to see? Artifact had lots of random events, but they were mostly small rather than adding up, so all that did was give better players unique situations to roll with and plan for. It had lots of decisions with long term payoffs, that compounded and played in to each other. Poorer players didn’t even notice many of the things that were causing them to lose. And this all plays into reason three.

Reason 3: Artifact Wasn’t Random Enough but Was Perceived as Random in Bad Ways

Randomness is vital to games. The biggest benefits of randomness are variety of game play and game situations, such that the game feels fresh, the opportunity for tension and surprise, the chance for players to beat players who are better than them so worse players so more matchups remain interesting and poor players do not lose interest, and giving players the chance to make interesting strategic choices where they have to decide what risks are acceptable and worthwhile.

As Garfield and Skaff point out in their textbook, luck and skill are not opposites. But Artifact definitely had (if anything) too much skill, and did not have enough luck. After a while, the games start to bleed together, following similar patterns. There are lots of little random events that can be good or bad for you, but those even out, so the better player reliably crushed the weaker player, more than in any other known card game.

The randomness the game did have was great. Knowing when to risk a bad arrow, or hope for a good one, and how to maximize your chances of getting one, was quite interesting, as was trying to maximize your chances of lining up well with where heroes would end up within a lane, or what would happen based on where the enemy placed their heroes versus where you would place yours, or your chance of getting the item you wanted out of your store, and so forth.

The problem is that if you add up a lot of little random things, you get far less than additive amounts of net randomness in the result. But you might get more than additive amounts of perceived randomness. Players see each individual random element as part of a constant barrage. And the randomness that Artifact did have was too often the kind of explicit non-card-draw randomness that creates player ire and complaints.

Mark Rosewater and others who work on Magic often note that players react to different kinds of randomness in different ways. Coin flips and die rolls are the worst, when one outcome is clearly good for a player and the other bad, because they stick out as explicit intentional randomness. There is a reason we all hated Frenetic Efreet, and why we don’t make cards like that anymore.

The arrows of Artifact felt to players like that kind of randomness. As opposed to drawing different cards in the wrong order or wrong place, or having the tools for the wrong matchup, which are forms of randomness people accept.

The distributions of heroes, and who lined up against who, especially in the opening, was the same. Sometimes you would have heroes face off and someone would go back to the fountain on turn one. Other times, they’d miss each other entirely. That was a big swing that players had little control over, although they could build their decks to assert more or less control over it. The secret was that losing those flips was much less bad than it appeared, since the hero comes back two turns later to the lane of your choice and not that much damage gets dealt to the tower. It definitely risks a runaway via item purchases, or perhaps very quick tower kills, but it’s quite manageable by design. Thus, it didn’t actually let bad players beat good players that often. But it sure felt like it did.

The lesson is to do your best to hide your randomness where it isn’t viewed as the bad kind of random, and to make sure it is available in at least some large chunks. Artifact’s best randomness was the placement of heroes simultaneously, because it doesn’t feel like randomness. The shop items also felt ‘fair’ because they were drawing cards from a deck.

Reason 4: Artifact Had No Meaningful Outer Loop

When Artifact launched, you could enter a casual queue and play matches that meant nothing. Or, you could enter a competitive queue that cost $1 (one ticket, if you played keeper draft you had to bring packs as well) and returned slightly less than that in value. That seems like the actual minimum charge required to keep the matches interesting without embracing toxic free-to-play style mechanics that would have destroyed card value along with the overall game experience. So hats off to them for charging just enough to keep it interesting, and not a cent more. The problem was that this was all you had to do, unless you found a tournament online that would last all day and was definitely a niche product. Winning at first did not bring any benefits whatsoever, beyond the small return from the tournament itself. The only benefit of winning was you played against harder opponents, but your rating was invisible to you, so you got to experience the frustration of not winning more when you got better at the game without the reward of knowing that you were improving via the rating. There were no larger prizes or thresholds or other things to aspire to, beyond what the community created for itself. Later, Artifact added experience points and levels in a tacked-on fashion, parallel to the ratings that actually mattered and continued to remain hidden. The new system meant that if you played games, you had a number that slowly went up, which slowly gave you a small number of packs and icon rewards, and allowed you to have a higher number next to your name. It was really weird that I could know how many games my opponent played, but not how good they were in any real fashion. This seemed to not fit a game that was so much about skill. There was no meaningful ‘collect cards’ outer loop, because drafting cards was not a practical way to assemble what you needed – any reasonable person would simply buy what they wanted, especially once the prices started coming down. There was no meaningful achievement loop. When I thought about playing games, I did not expect to get any form of reward beyond the game itself, other than the opportunity to improve my skills. In today’s world, that is not acceptable. What should Artifact’s outer loop have been? At a minimum, Artifact could have had a system that allowed qualification towards its big prize tournaments. Given Artifact’s emphasis on skill, giving players a competitive structure to climb seems like the obvious thing to do, and was something that was promised in the form of a million dollar tournament that was never delivered – which at least to me severely damages the credibility of future Valve promises. Artifact should also have given players the chance to know their true Elo ratings. What they did offer, in the form of cumulative rewards, was much better than nothing, but the rewards quickly became few and far between in a way that felt super far away, leaving players without realistic aspirations, and strongly encouraged players to play only a small number of games before mostly maxing out. Without getting into the weeds here too much, there are a lot of ways to vastly improve how much such a system encourages play. Another key part of the outer loop is ‘keeping up’ with expansions. When one plays in Arena, Hearthstone or Eternal, one is constantly aware that one’s collection naturally decays over time, with continuous play required to get the cards to keep going. The only alternative is payment. Artifact of course was planning to have expansions, but not even one of them was ever announced. This was an underappreciated huge deal. Without knowledge of a future expansion, the most important and unique outer loop of all was crippled – my collection was complete and would remain so for the indefinite future. Reason 5: Artifact Had No Good Budget Format What is so weird is that they ran cash prize tournaments in a very good budget format, called Commons Only, during the beta. This format spontaneously turned out to foster interesting play that focused on lane resource allocation, as playing only commons means that mostly things in a given lane remain stuck in that lane. This is an excellent way to get one’s feet wet, as the game does not need to be grasped as a unified whole in the same way. It had several viable decks at the highest level that covered the four colors. Then for some reason, this did not become a default format to offer players. I still have no idea why, beyond fear that players might not feel enough need to buy more cards, but I don’t model Valve as thinking that way. Without a budget format that could sustain itself, the game seemed much more expensive and off-putting, as playing Artifact with inferior tools can be very disheartening. This then made Artifact’s economic model seem and be worse than it would have been otherwise, making one of their biggest problems even worse. Reason 6: Artifact’s Economic Model Was Marketed Poorly and Misconstrued I put this before the important real flaws in the economic model because I believe that perception of Artifact’s model as greedy, stingy, unreasonable and out of step with the times was far more hurtful than the real flaws it had. Artifact had in effect a perfectly reasonable economic model, but players did not see it that way. What happened? A lot of it was the way the model was presented. Artifact did not highlight its customer-friendly economic features. Artifact charged minimal amounts for tournament and draft participation, even allowing you to play for free if you chose to do so, at the expense of your opponents not having a strong incentive to give the games their full attention. When entry was charged, the rake taken on it was very small. This is a big deal. In all the other collectible card games I know, playing limited is something you earn through grinding, or something you buy at remarkably high hourly rates. In Artifact, once you pay the initial$20 fee, you can draft as many times as you want, for free, forever.

That’s the best drafting deal ever offered, and that is how Artifact’s offering should have been framed. It is also how the bulk of players should have been playing the bulk of their games.

If players had thought of Artifact as a $20 experience that allowed you to play its primary mode of play forever, and also gave you assets you could sell to get a lot of that$20 back (in the first week you could get more than all of that $20 back, although that did not last) then players would have seen that this is a good deal the same way that paying$30 for Slay the Spire or $60 for Dragon Quest XI is a good deal. You then have an option to play constructed, which can cost up to a few hundred for a complete set of playable cards, but there is no one forcing you to do that if it is not interesting to you. The cost of constructed in Artifact started around$200, then decreased as the game failed, but $200 for a full set of playable cards from a large set compares favorably to what you would pay in other similar games. It compares even better if you consider that you can sell those cards afterwards and hope to potentially even profit, and you can choose to buy only those cards you need, without having to go through a super inefficient dusting or wildcard system. But none of that mattered, because all people heard was ‘Artifact is not free to play.’ This cached out as two things: Artifact costs$20 initially (whether or not this was at first effectively 0 given that the cards you get along with that can be sold) and Artifact does not reward your daily play with cards that suffice to let you build a constructed deck. Players only considered the question of ‘how generous are you to people who don’t want to ever pay money but do want to build a collection’ and generally looked for markers of generosity within the free-to-play business model I consider deeply flawed, rather than considering the system holistically or comparing it to a normal pay-to-play title. Thus, players got angry, and the game got review bombed, and professional reviews constantly referred to the game as greedy and as using an aggressive, outdated business model. Once that becomes the story, you’re in deep trouble, whether or not it is accurate. Reason 7: Artifact’s Economic Model Was Flawed As much as I will defend Artifact’s economic model as being unfairly maligned and having the right overall concepts at its foundation, the execution of those ideas was botched. One of the ways this happened was very not subtle and should have been obvious, especially since I tried to caution them about it. The others were understandable mistakes that would have been difficult to notice in advance. The big mistake was to lock all transactions to the Steam Marketplace, preventing players from transacting with each other directly, and then take a 15% cut of every transaction. 15% is a freaking huge fee, and the desire to collect that fee ruled out any ability to loan cards or give cards to friends. Thus, the ability to let a friend play a deck you’ve assembled, or try out something cool, was gone. Players ended up feeling tied to their cards a large percentage of the amount they would have felt tied with no ability to sell at all. One of the big advantages of having cards that can transfer is the ability to recapture value, but a 15% transaction tax makes that story far more difficult to tell. Another is to have tight markets so you feel like you’re getting a fair price when you buy or sell. Magic Online provides this through the use of trading bots, so you don’t feel like trading is too expensive. The problem is that if every card comes with a 15% fee, then that makes any profitable trading strategy need to widen out by 15% on top of the 15% fee, so they can pay the 15% when it’s their turn to pay it. So effectively there is a 30% gap between buy and sell prices, plus the profits to the dealers who are worried their inventory will lose value and at paying large costs to get rid of things they no longer want, so you’re looking at a gap of 40% minimum when you don’t have ‘natural’ flow to trade against. That is super punishing. Combine that with an interface for trading where it suggests prices you might want to seek to pay, but which often fail to actually make a trade happen, and the whole experience is super frustrating. I felt bad every time I went to use the market for a non-trivial purpose, and all my dreams of speculation or investment in undervalued cards went away completely the moment I saw the market structure. Even worse was that the game was creating sources of surplus card flow to be dumped onto the market, debasing the value of the cards. Starting a new account forced players to open20 of product, without giving most people much reason to keep those cards, since they would end up mostly playing limited or leaving the game entirely. So why wouldn’t they reclaim what value they could by dumping everything into the market, even at a big loss? Then, when this flow pushed prices down, players saw cards as an increasingly bad investment, further depressing demand, and accelerating the downward price spiral set in motion by a declining player base.

We add to that the giving away of product in daily rewards. The daily rewards may not have looked like much to players, but it was still a large flooding of the market, because many of the players getting the free packs had no reason to hang onto them when they could get some steam credit by dumping the cards. I’ve been working on modeling this interaction for some time, and I firmly believe that you mostly must choose two of these three: either give cards away steadily for free, or allow cards to be exchanged freely, or allow cards to retain value. If you give away cards and let them be exchanged, the market floods and value is depressed. If you give away cards and want them to retain value, you have to lock those cards down (if not all cards) to avoid this. If you don’t give away cards for free, then value is maintained because incoming supply is always paid for, so unless you have a large decline in the player base, someone is paying for that supply at whatever price you charge.

If you’re going to have a marketplace and want your cards to have value, it imposes real restrictions on you. What you distribute has to be chosen very carefully. Players need to fully own their cards and be able to reasonably transfer them. Players need assurances that their card value will be protected.

You have to mean it, or it won’t work.

Reason 8: Artifact Set Expectations Too High

Artifact came from Valve and Richard Garfield, based upon DOTA 2, looking better and with more polish than any other digital card game has ever launched with by a wide margin. The game is gorgeous, the result of years of loving detail and care, and came after a long gap of game publishing from a studio with a long record of hits. It was presented as the Next Big Thing, and a lot of people got super excited. Tons of players tried it out when it launched.

Alas, a lot of those players had no business playing Artifact, because they weren’t the type of player who enjoys that level of complex strategic interaction. And also alas, the game launched essentially unfinished, because it didn’t have its outer loops in place in any form, and without a good way to bring new players up to speed, but it was presented as fully ready rather than early access.

This resulted in a strong early peak in users, followed by a huge decline, which looked like failure and also did terrible things to the market, and resulted in terrible reviews because people were evaluating the game against a standard it never had a hope of meeting.

If Artifact had launched first as early access, allowing players to first view it as a game in progress, all of that could have been avoided. It also would have allowed them to fix their problems before it was too late, because they would have avoided the next issue.

Reason 9: Artifact Did Not Run the Right Tests Because They Didn’t Test Their Economy

Artifact ran an extensive beta test where many of us played a lot of games. The problem was that what they didn’t do was test Artifact where players did not have full card access. Throughout the beta, players were given complete collections.

Which was super awesome for the players, and allowed us to better test the ‘final form’ of the constructed format. What it didn’t do was give an idea of how players would in practice interact with the game. What would their experience be? What would they choose to do, what types of decks would they build and face? Would the economy work? How would desire to build a collection interact with and drive player spend and behavior? And so on.

These are super important questions. Artifact’s test of them was on launch day.

Arena solved this problem by running a beta with its full economy working as designed, with credits from card purchases being given when the cards were reset later. That solution is not available to a game with a full marketplace, especially if you want to predict player actions and market dynamics, so the problem isn’t a trivial one to solve. If I buy a copy of Drow Ranger in the market, someone gets that money, so Valve can’t refund it without taking those proceeds away.

My guess is that would indeed be the right thing to do – to take those profits away upon reset. For the test period, cards and packs cost Relics which can be bought with steam credits, and the market trades in Relics. At the reset, Relics are destroyed, and all funds used to purchase Relics are returned as either steam credits or as funds one can use to purchase selected items in Artifact.

The important task is to impose some set of reasonable card access restrictions, and see how players build decks, play the game and use the market in response to that.

Reason 10: Artifact Did Not React Well or Quickly to Initial Failures and Did Almost No Press

It is normal for a game to see a huge decline in play numbers over its first few weeks. No one knows that better than Valve. So it was understandable for Valve not to be too concerned about the decline in numbers over the first few weeks, until it was clear that the ongoing trajectory was not going to improve. More troubling should have been the reviews of various types that were coming in, the reports of players dropping the game or not streaming or watching the game due to complexity concerns, and other not-the-number-of-users red flags.

What initial reactions they did have, enshrining free drafting and giving a first draft of player achievements and rewards for playing, were good starts, but did not turn around the narrative. Valve counted, as it has done in other cases, on its players to get the word out. That didn’t work, and players continued to view the economy as greedy and the game as having nothing for them to do.

Still, those stopgaps combined with the excellent reception the game got from many lovers of deep strategy games, bought Valve some time. During that time, they could have made further improvements, offered more things to do, or at least announced future things and improvements that players could anticipate. They also did not engage in any public interactions that showed they were aware of the issues, or illustrated their thinking or intentions.

Also importantly, for a game whose core success strategy was supposed to involve top level competition, they left actual competitions entirely in the hands of outsiders. There were some cool tournaments held, but Valve itself didn’t have any planned or offered. Players who came to the game without knowing about the tournaments didn’t realize there was any organized play. What play was organized didn’t have a clear path to getting involved other than either knowing someone or signing up for a single elimination all day tournament through discord. When I say all day, I mean all day. I think giving us a few Grand Prix style events, with five figure prize pools, that could be qualified for via wins or win rates in queued events, would have gone a long way towards tiding things over. Having that feed into a fully announced and scheduled million dollar first prize tournament? Even better.

Instead, every week that went by we all wondered whether Valve would be willing to follow through, or would instead abandon its pledge. Which is what eventually happened – it turned out to be a conditional promise that had assumed the game’s success. Why were we still playing?

When the time came to address the metagame being stale, and people complaining about some cards being better than others, Valve responded not by printing or releasing new cards or expansions, but by changing cards via fiat. This got them a lot of initial good press they had at least done something to respond to players and shake the game up. I saw that response, and thought that perhaps my negative reaction had been premature. If they kept changing things to keep the game dynamic, and embraced this new system, that would be tough on card value and make it especially taxing on players who tried to take breaks from the game, but they would be providing new and different experiences, show responsiveness to the player base, and be able to iterate towards where they needed to be while we waited for the first expansion.

Alas, such iterated changes did not come, nor did more radical changes or any announced plans. When the first set of changes did not revive the player base on its own, that was it. So we never got to find out what that experiment would have looked like, or whether it would have worked. Nor did we try releasing the first expansion to see if that helped shake things up and make people think of the game as a changing and evolving thing.

I respect the hell out of ‘it will be released when it is ready’ as a principle of game creation. Kudos to those with the resources and determination to pull it off. But once one releases a game that requires continuous play and a critical mass of players, one no longer has the luxury of time. One certainly does not have the luxury of time with no communications.

Conclusion

Artifact was, as my review made clear, a uniquely amazing game in important ways. It did a lot of things uniquely right. Alas, it also did a lot of things uniquely wrong. I’ve listed ten. If we could fix one of the ten, could we have saved the game? My guess is that no one of them alone would have been enough to turn the game into a success if it were fixed, but each of them would have made a substantial difference. Solving a lot of them, I believe, would have been enough. While the complexity issue in particular is a huge deal, that complexity also brought many benefits, and I think that with proper handling of other issues the complexity problem could have been navigated, as it was in the original DOTA 2. It was the combined weight of so many different problems that ended up bringing the game down.

Could the game still be saved? There is a bad taste in everyone’s mouth, legacy economic problems that will weigh on the enterprise, and the players have almost all checked out. At this point, time pressure is no longer a major concern, as everything that can be lost to delays has already been lost. If the game came back in a few years, with solutions for its problems and proper support including prize support, I do believe it can still be a modest success on its own terms. I do not expect that to happen.

The greater tragedy would be to draw broad conclusions about what is fatal to a game, and for Artifact to become a way to inhibit innovation and drive games towards becoming Hearthstone clones (or otherwise stick to existing templates) even more than they already do. The more we realize how many different ways there are to improve our prospects when exploring interesting new space, the better our chances for creating the next unique great thing.

Discuss

### When is pair-programming superior to regular programming?

8 октября, 2019 - 14:48
Published on October 8, 2019 11:48 AM UTC

When doing software development there are many choices that can affect your productivity. There are choices about how a company works. One of those is doing pair-programming over doing solo-programming. We recently had a consultant at our company who claimed that pair-programming is proven to be better but who couldn't point to evidence.

How does the case for pair-programming look like? What do we know? How can we rationally think about the question?

Discuss