## Вы здесь

### A Key Power of the President is to Coordinate the Execution of Existing Concrete Plans

Новости LessWrong.com - 16 июля, 2019 - 08:06
Published on July 16, 2019 5:06 AM UTC

I listened to the 80k podcast with Tom Kalil, who spent 16 years as Deputy Director of the Office of Science and Technology Policy at the White House. Kalil seems highly competent at evaluating concrete scientific plans to offer the president and finding the path of least resistance through government to effect those plans, though not a generator of new scientific ideas, nor someone with deep technical understanding of any single domain.

One key idea I took from the podcast was that his main use of the executive branch of government is as a coordination mechanism. I moved away from thinking of the President as an expert who makes decisions, and much more as an individual with immense coordination power trying his best to take any concrete plans given to him and coordinate the country around executing on them. That is, not someone who comes up with plans, not someone who executes on the plans, but someone who coordinates people to execute the concrete plans that are waiting to be picked up and run with.

Below are relevant and very interesting quotes, followed by a few more updates I made listening to the podcast.

Kalil also talks about in his role as Deputy Director of the Office of Science and Technology Policy, he helped raise the staff count from 40 to over 100 during the Obama administration. He just gets to hire people who are excited about an idea and want to make it happen, and then they make it happen using the coordination power of the executive office. Here's a prominent example:

Tom Kalil: Let me give you one example. A young woman emailed me and the subject line of your email was, “Cass Sunstein says I should work for you.”Robert Wiblin: That’s a strong subject line.Tom Kalil: Good subject line. So I did a little research on her. It turned out that she had been a child violin prodigy with Itzhak Perlman, had won the major Yale undergraduate awards, was a Rhodes scholar, and was wrapping up a post-doc at Stanford in Decision Neuroscience. I went out on a limb and I decided to take a chance on her. Her name was Maya Shankar. I asked Maya, “What do you want to do?”Tom Kalil: She said, “The UK has created this organization called the Behavioral Insights Team, which is taking these insights from people like Kahneman and Tversky and Sunstein and Thaler and using them to inform policies and programs. These are all US researchers. Why don’t we have something like this?” She said, “I would like to create that.”Tom Kalil: Sure enough, in her late twenties, she arrived with no money, created this new organization called the Social and Behavioral Sciences Team, recruited 20 behavioral scientists to the federal government, got them to launch 60 collaborations with federal departments and agencies and got President Obama to sign an executive order institutionalizing this new entity.Tom Kalil: I think that’s pretty consequential for someone in their late twenties to be able to accomplish. That’s one thing I did, was to recruit people of that caliber and teach them how to get things done in the federal government because the government doesn’t come with an operating manual.

The next quote is about how the core goal of the office of science and technology policy is to take the necessary steps to get the private sector to build new tech:

Tom Kalil: One of the things I learned is that if the United States is behind in a technology, it’s very difficult to try to re-establish a leadership position… We tried to do that in the area of technologies like flat panel displays and we invested some money, but I don’t think a whole lot came out of it.Tom Kalil: Once Korea and Japan dominated the market for things like active matrix, liquid crystal displays, then trying to get the United States back into that market is really, really hard, and might require more money than the US is willing to put into it. Because obviously we believe that the primary role of government is to create the right environment for the private sector. It’s not to engage in this sort of heavy handed-top down industrial policy that you see a China engaging in, for example… we invested in this idea of flexible electronics where the idea is – maybe you have a display that’s a piece of paper that you can roll up and put into your pocket. And, if that’s an area where no one has established a clear leadership position, that’s more likely to be effective than saying, okay, we’re going to duke it out in some market that we’ve kind of already lost.

This final quote is an example of the coordination power of the President.

There is a lot more genuinely fascinating discussion in the interview, especially Kalil's comments on using financial prizes to incentivise science+tech in areas like education and poverty.

My new model is that the President's interaction with science is largely to take concrete ideas floating around in the environment that are ready for their time, and push them over the edge into actually being built by the US private sector, or into actually substantially informing government policy. This is similar to the notion that scientific ideas come about when the environment is ready for them (Newton and Leibniz both discovering calculus at the same time). There are executable plans floating around in the ether, and the President keeps getting handed them and sets them off. His department is not an originator of new ideas, it coordinates the execution of existing ones. (And this does seem obviously the correct marginal use of attention from the President. Compare 15 minutes per project versus spending a week becoming an expert in one and then executing it himself.)

I’ve updated positively on the tractability of gaining influence within the government and being able to use it on timescales of 4-8 years. (I expect I will likely make a further update when I read the blogposts of Dominic Cummings regarding UK politics, though not sure how strongly.) Overall I think influence in government, if you’re ambitious and well-connected and have a very concrete vision, is likely quite a real action one can take. I expect that from the perspective of government there is a lot of low hanging fruit to be picked.

I updated negatively on the usefulness of interacting with President and his environment in the short-to-medium term. My sense is that the state of understanding of how transformative AI will be built and what impact it will have on the world is sufficiently low resolution and confused that we have no project or policy recommendations for the government, and will not be able to do so until we see further work that helps conceptualise this space. Listening to the podcast tells me that if you get 15 minutes to talk to the President about x-risk today, you are wasting his time, because we have no concrete plan that needs executing if only could coordinate major AI tech companies. We have no R&D projects that need funding. We have no nuanced AI-development policies for global powers to agree to. I’m pretty sure that there are people in this community who can coordinate Elon Musk and Demis Hassabis or whomever else, should we have an actionable plan, but the current state is that we have no plan to offer.

Discuss

### How Can Rationalists Join Other Communities Interested in Truth-Seeking?

Новости LessWrong.com - 16 июля, 2019 - 06:29
Published on July 16, 2019 3:29 AM UTC

Last week, Davis Kingsley wrote about reasons one might want to diversify one's "friendship portfolio." I commented that if this, being on LessWrong, was aimed at a rationalist audience, it's my experience many rationalists are introverted or shy enough, some have difficulty joining new groups of friends outside the rationality community as well (though of course there were several comments on Davis' posts about the benefits of having just a small, consistent group of friends in a single community, worth considering). However, Ruby commented with an even greater reason why some rationalists who might want friends beyond the rationality community nonetheless primarily stick to the rationality community:

I suspect there are challenges for rationalists in joining new communities beyond introversion. I've found it jarring to be getting along with some new folk and then people start saying ridiculous things, but worse, having no real interest in determining whether the things they say are actually true. Or even when they try, being terrible at discussion. I don't need to nitpick everything or correct every "wrong" thing I hear, but it is hard to feel like beliefs aren't real to people - they're just things you say. A performance.There are people outside the rationality community who are fine at the above, but being used to rationalists does introduce some novel challenges. It'd be nice if we ever accumulated communal knowledge on how to bridge such cultural gaps.

So, I thought I would ask.

Discuss

### Interview With Diana Fleischman and Geoffrey Miller

Новости LessWrong.com - 16 июля, 2019 - 04:34
Published on July 16, 2019 1:34 AM UTC

I had the pleasure of interviewing Diana Fleischman and Geoffrey Miller at the NYC Rationality meetup. Diana and Geoffrey are professors of evolutionary psychology, Effective Altruists, thoughtful polyamorists, and fearless thinkers. We talked about everything that’s important in life: gems, sex, morality, kids, shit-testing, jealousy, and why women are smart.

If you missed it, here’s the wide-ranging interview I did with Geoffrey alone a year ago. I will also publish the audience Q&A from the meetup later this month.

Congratulations on your engagement! Geoffrey designed a special moissanite engagement ring for Diana, and moissanite is something my wife and I believe in very much as well. Can you tell everyone about moissanite so that no one here buys a diamond ever again?

Geoffrey: In my book Spent I did a multi-page critique of the DeBeers diamond cartel and diamonds as costly signals of commitment. And maybe it’s fine to spend a lot of money on a signal, money that goes to a cartel.

Diana: But cocaine is more fun!

Geoffrey: Yes, or get a college degree — that’s a cartel too. But there are better and cheaper stones, like moissanite which is a silicon carbide gem and costs 1/10 to 1/50 of a diamond depending on the carat. But I think it’s a better stone, it has superior ‘fire’ and ‘brilliance’, as they say. Of course, once moissanite was invented and it became colorless and clear enough, the diamond cartel dissed it by saying “these are just disco balls”. Basically – it’s too flashy, and no self-respecting person who cares about the subtlety of a diamond will buy a moissy. Whereas for the previous 100 years, diamonds were sold as the flashiest, highest brilliance, and highest fire gemstone. So: hypocrites.

Moissies are getting better and better, and they just went off-patent four years ago so there are multiple manufacturers making them now and prices are dropping. So I’d rather put the effort into the design and personalization. I’m using a CAD system for the design and manufacture, you can now design it in collaboration with a jeweler who will cast it and make it.

Diana: It’s about a 9 mm stone. How much would a diamond that size cost?

Geoffrey: It’s a 3-carat moissy so it costs about $2,000, and an equivalent diamond would cost$120,000.

My wife’s moissanite sparkling in the beerlight.

Geoffrey wrote the book Mate with Tucker Max with dating advice for men and in 2015 they had a podcast called The Mating Grounds that Diana went on before she was dating Geoffrey…

Diana: Actually, I was already dating him. When I was on the podcast it was the end of a two-day visit, the first time I visited Geoffrey. But then on the podcast, Tucker didn’t want to make that obvious.

Geoffrey: So we were in the afterglow of the first blush of romance, and Tucker wanted a very serious podcast about dating advice for young men.

Diana: I actively pursued Geoffrey which is good, because the women that Geoffrey pursues are all mentally ill.

Geoffrey: That’s largely a true statement.

Diana: Yeah, it’s practically diagnostic.

I tried to chat Geoffrey up in 2012, soon after his divorce. And my trick at the time for chatting up people who are introverted and quiet was to disclose more and more embarrassing things about myself until I got a reaction. So I was trying to draw him out, and he was like a clam in his shell. So I said, “Well, you must have plenty of other people to talk to, I’ll let you be.” And he answered, “No, no. This is fine.”

Geoffrey: I was scared you would go away at that point. Being aspie and introverted I thought I was giving lots of signals of interest.

Diana: You were not giving me signals of interest. You weren’t even properly facing me.

Anyway, after that point, I was messaging a lot and I arranged that visit to Austin to see him.

Then I told Geoffrey I lived a long way away and I had another boyfriend, who by the way was very supportive of me dating Geoffrey because he likes Geoffrey’s books. This is something only poly people understand — the great networking opportunities you get when you suggest hanging out with someone because you both like the same girl.

That was also Geoffrey’s first foray into poly and he was a bit confused why he’s hanging out with this woman with her boyfriend.

Geoffrey: I just assumed, based on the evolutionary psychology I had learned to that point, that we would meet for brunch and then fisticuffs would ensue. Maybe he would bring sabers to defend your honor.

Diana: And after that, I told Geoffrey that he should get a local girlfriend and helped him get on OkCupid. And then after a couple of years, we became more heavily involved. I didn’t really want a long-distance relationship, and now I’m moving to Albuquerque in July [where Geoffrey lives].

You mentioned somewhere that you first fell in love with Geoffrey after reading his book, The Mating Mind.

Diana: That’s true.

I have a question to Geoffrey here. According to evolutionary psychology, a lot of what we do, including intellectual things like writing a book, is deep down motivated by trying to get laid. And after studying evolutionary psychology I’ve noticed it in myself, that every blog post I write or meetup I organize is in some small part about how attractive women will look at me. Do you enjoy having this as a motivation or are you trying to fight it and do things for other reasons?

Geoffrey: I sort of embrace it. It’s important especially for young people to understand that an awful lot of their behavior — self-improvement, moralizing, politics, and ideology — is covert mating effort to some degree. But the superordinate category is trait signaling – signaling something to someone. It’s not the case that all human creativity is just mating effort. A lot of it is status seeking, trying to impress other people like your parents, friends, and neighbors. The signaling principles are the same: doing something difficult that’s intellectually demanding, artistic, musical or funny tends to work for mating and also the other social signaling functions.

I think, like Robin Hanson and others, that if you’re tuned into signaling that’s really helpful. And when I accuse people on Twitter of woke virtue-signaling or SJW signaling I’m being derogatory not because I think that virtue-signaling itself is bad but doing it in a certain way, or unselfconsciously, is bad. If you do signaling with a lot of self-awareness that’s totally fine and it’s at the root of a lot of human progress and creativity.

Diana: I think it’s good that men do so many things to impress women. Geoffrey’s thesis in The Mating Mind is that we’re a lot smarter than what we need for survival, and the human mind is a lot like the peacock’s tail. It’s a result of runaway selection, and we’re showing off ostentatiously how smart we are. And women are smart to be able to evaluate intelligence and men’s creativity, so they became as sophisticated.

Geoffrey: That’s actually the wrong model, which I rejected.

Geoffrey: You may want to reread it again. [smiles]

To be clear, the question is: why are women smart? That’s a great question.

Geoffrey: I do consider, in chapter 3 I think, whether it’s just a standard runaway process like the peacock’s tail with females doing all the choice and males doing all the display. I ended up saying, because the difference in brain size is actually very small and the difference in general intelligence between the sexes is negligible, and there are no major differences in domains like language, art, and music ability — both men and women can sing and paint and do comedy — I ended up with a model of mutual mate choice. Both sexes are displaying to some degree, sometimes men display harder and sometimes women, especially once in a relationship that they want to maintain. That’s the model I ended up with, that both sexes choose and both sexes display, and that is why there’s low dimorphism between the sexes in human mental traits.

Diana: In terms of status signaling, it’s good that in different subcultures and places around the world status can be gained in different ways. Otherwise, men would just be doing what gets the most female attention. You’re writing blogs and hosting rationalist meetups where the sex ratio of your audience probably skews quite male, and it’s good that men are doing things to impress not just mates.

To be fair, I haven’t actually gotten even one date as a result of my blog. But deep in my animal brain, every time I submit a post I’m thinking, “This is the one that’s going to do it!”

Diana: Women are attracted to status, it’s called hypergamy, and luckily the signal is noisy otherwise we would be doing a more circumscribed version of what we do to get laid. But now there’s a huge variety in the things we do to seek status and signal status.

One thing we do to impress people is altruism. Now Bret Weinstein talks about how once we understand how things like terrorism and genocide are evolved adaptive behaviors, we could overcome them and stop killing the outgroup or whatever. So I’m worried about how that applies to altruism. Spencer Greenberg gave a talk at EAG about introspecting to find our intrinsic values. But I worry that if I introspect I’ll find that I donate to GiveDirectlybecause there are attractive women and cool people at Effective Altruism meetups.

You wrote an article about how our evolved moral intuitions stand in the way of a “higher morality”. So where does that higher morality come from?

Diana: I’m unabashedly utilitarian. It comes from my experience with pleasure and pain, and then just extending that out with rationality and computation. But that still needs to be leveraged.

In 2014 I went to my first Effective Altruism conference and I’ve been hearing a lot about existential risk and how important it was to protect the future of humanity. I saw a talk by Eliezer Yudkowsky about this topic and how many trillions of lives there could be in the future. But I didn’t get it, and I didn’t like it.

And then I did a circling exercise, where you look into someone’s eyes and think of them as a person that you know and love. Then you circle again and think of them as a person on the other side of the world who is very poor, and then again where it’s a person in the far future who would like to exist. And then everything synced up and I became an advocate for working on existential risk.

So there are ways to leverage rationality to get to morality but in that particular case, I had to get my scope-insensitivity turned off by making eye contact with a particular person to care about that issue.

We should do this exercise at a meetup!

Going back to relationships. I just finished reading The State of Affairs by Esther Perel. She talks about how if you want to get good at business or sport or any skill or craft there are a million mentors and role models you can learn from. But it’s very hard to find role models for relationships.

Most people don’t know many others who have great relationships, and even those that seem to have great relationships aren’t as open about what they do as athletes are about their training regime, for example. Fiction could be another source, but what people are watching are shows like Game of Thrones which isn’t really full of healthy marriage advice.

Who are your relationship role models, and do you consider yourselves fit to be relationship role models?

Geoffrey: The second answer is that we aspire to be good role models for managing relationships. We don’t divulge every last detail of what we do, but we do share our tips and tactics and we’re both working on books on how we manage things even in high stress situations like being long distance for five years. Or being poly and open.

In terms of role models, I grew up with a lot of respect for my parents who got along really well 80-90% of the time. I didn’t see how they managed their conflict resolution but they seemed capable of a fairly high degree of rationality and they got over tension fairly quickly, in a matter of hours rather than days or weeks. So I saw that it’s possible to at least have a reasonably successful marriage. My mom and dad also cooperated effectively not just on raising kids but also on local political issues and jointly managing a business so I also saw them as a team.

I asked people about good relationship role models on TV and the only one I came up with was Madam Secretary. Tea Leoni plays a high-powered CIA operative turned Secretary of State and her husband is a professor of moral philosophy and ex-marine who helps raise their three kids. And the screenwriters can afford to make their relationship work because all the drama is external to the marriage. It’s either tension with Iraq or tension with their teenage kids, not between the spouses.

This was the only example of screenwriting that had a good marriage with efficient communication that was also still hot and sexy. So if anyone has other ideas, email me.

The McCord family on Madam Secretary, CBS

Diana: Those people are psychopaths. They’re a poly couple, but they also kill people.

Killing together really binds a couple.

Diana: It really does, murder and threesomes are the keys to marital success. In Breaking Bad there are also periods where their marriage worked really well, like when the wife breaks bad with him.

I was also trying to think of lesbian and gay relationships because the media is very lefty so they may be more inclined to represent a gay couple as better functioning. But I couldn’t even think of one there.

For my part, I’m incredibly cynical about my own motivations. Geoffrey and I just got back together yesterday after not seeing each other for five weeks. A couple of interesting things happen because of being long-distance that I perceive through an evolutionary lens. Right before I leave there’s some mutual punishment happening for the imminent departure of the other person. Slights and any signs of emotional withdrawal are taken much more seriously. And then when we see each other I’m really happy to see him but there are also a couple of days of ‘shit testing’. Shit testing is the red-pill idea that women are being difficult in order to test the status, fortitude, and commitment of their partner. So I’m giving him shit to re-test how well we get along after a hiatus.

In some sense, I’m making fun of myself, but today I got really angry about a tiny thing where he didn’t hear me properly. And I was calling him names even as I was laughing about it. It takes a kind of cynicism about yourself and your motivations to have a good relationship. You have to realize these evolutionary things, and also to really want to make the relationship work. That’s what I hope to bring in the book I’m writing: if you can accept that as a woman you want to completely control and monopolize a man, that you have these desires that are selfish and destructive, you can achieve better outcomes.

So the secret to relationships is either to commit crimes together or commit crimes against each other but laugh about it.

Geoffrey: I’ve been teaching a course on human emotions for 10 years, and it’s really important to get each group of undergrads to look at their emotions at arm’s length. To think: what are the origins of each emotion, what are its adaptive functions? And in a modern context – which residual functions are still relevant and which aren’t. You can do this with disgust, jealousy, anger, shame, guilt, pride, awe, everything.

And the students start to realize they don’t have to take their emotions that seriously. If you combine the evolutionary perspective with a mindfulness perspective you can take your thoughts and feelings as passing events that come and go, like Sam Harris talks about. Then it’s much easier to have good relationships.

Diana: It’s very hard to do with many emotions in relationships. For example, the jealousy or outrage you feel when you get a sense that your mate is divesting from you — ancestrally this was such a matter of life and death that anyone who could hold these emotions at arm’s length was at a serious disadvantage. And so today it’s hard for us to do so, compared to holding at arm’s length being angry with an acquaintance, for example.

But we know that smart people have better self-control and are also less likely to get divorced, they’re better able to negotiate ways to support each other. And you don’t see geniuses at relationships on television.

I want to ask how you think about jealousy. From an evolutionary psychology perspective, it seems like jealousy is somewhat obsolete. If it was meant to protect men from raising other men’s children and women being left without resources to survive, but we now have condoms, paternity testing, alimony, etc.

In the US, jealousy is taboo. Even people who get cheated on are supposed to be angry but not admit or talk about jealousy. But in Latin America, jealousy is embraced as a component of romantic relationships. Diana, you were born in Brazil and grew up in the US. What do you make of jealousy?

Diana: It’s unclear when jealousy comes online for children. David Buss has a paper arguing that potentially young men don’t really understand jealousy until their first romantic relationship. I didn’t understand it as a child. I remember my mom with her friends talking about what somebody was going to name their baby, and I suggested naming them after ex-partners because of how important they are. And I didn’t understand why everyone said it’s a terrible idea. And then I dated a guy who wanted to name his daughter Genevieve and I freaked out because I thought people would shorten it to Jenna which is his ex-girlfriend’s name.

It’s true that people see jealousy as a signal of your investment and passion for somebody. Certainly, my mother who is Portuguese thought that lack of jealousy means that two people aren’t right for each other. Whereas people from Scandinavia see it as something that just happens, not a signal of one’s love or commitment.

And I never saw my dad jealous about my mother. When I was 16 and we were at a Mexican resort and some guy groped me, my dad just said that it’s a thing that happens. I think that parents’ protectiveness of their children is also part of the same thing. As kids these days say, it’s the same energy.

Related to that, I just saw an article about bonobo mothers do everything to get their children laid including beating away other males when their sons are having sex. But humans seem to have a tendency to suppress the sexual activity of their siblings and children, instead of teaching them seduction techniques for example. This seems quite maladaptive. Why don’t humans encourage the sexuality of their relatives?

Diana: I think they do, but it’s oblique. In Latin cultures you do see that; I was dating a Colombian guy who had other women too, and when I was with him his mother was fielding calls from his other women. Likewise with my Portuguese cousin, whose mother lied to the four women he dated on the phone about his whereabouts so that he could keep having sex with them all. You do see this in Latin so-called “macho” cultures.

In terms of seduction techniques, people are weird about talking about the sexuality of children but this happens all the time. Someone could be ready to slap their kid and the child says, “Mommy, you look so pretty today” and it works. This is teaching children how to get someone out of a bad mood or get on someone’s good side with compliments, which is basically seduction. It’s a very useful skill.

I did a paper in graduate school about daughter-guarding. Parents are much more careful about watching their daughters, more likely to have curfews, tell them what to wear, etc. We also asked parents how upset they would be if they found their child was having sex; boy’s parents said on average they would be only slightly upset, -0.7 on a scale from -3 to 3. And girl’s parents would be very upset, basically -2.

But it seems like bonobo parents would say +3 for boys!

Diana: We also asked parents if they care who their child is dating. Fathers only care about their daughters’ mate choices, but mothers care about both.

There is a big difference between bonobos, chimps, and humans. Humans care about having a reputation for fidelity, and men want to monopolize a woman’s full reproductive potential rather than just focusing on whether she is fertile right now. Chimps prefer a female who has offspring already because that’s proof of her fertility, whereas humans, cultural differences aside, often want to monopolize someone before she has had any children.

Bonobo mother and child

Geoffrey: This is a fundamental thing that Sigmund Freud got wrong. It’s not that daughters actually want to sleep with their fathers, but each child wants to practice courtship and signaling on the opposite-sex parent to figure out what works. Or practice on the opposite-sex sibling. So when kids are being cute or funny or showing off, they’re not actually trying to sleep with their mom or step-sister but rather using their feedback to calibrate and develop courtship skills they’ll use later as adults.

But it is a weird thing that in America dads aren’t usually giving explicit courtship advice to sons. Sons miss it and resent not getting it.

Diana: Obviously there are no studies of this courtship hypothesis, you can’t get approval for studies that involve children and sexuality. But I did hear a talk a while back that said that the best indication of how good a kid is at lying is whether or not their parents use corporal punishment. There’s a genetic component there, parents who hit children may also have some Dark Triad traits themselves, but kids learn to lie really well if they get physically assaulted otherwise.

So when parents hit their kids, they’re training the kids to be better at deceiving the parents, and other people too. So perhaps they’re not trying to get their kids to be obedient but to make their kids better at fooling people.

Speaking of interesting parenting techniques: as two evolutionary psychologists who fight against the blank slate theory and know about signaling, how will you raise your kids? Most of us in this room know the basic stuff that modern parenting gets wrong, like Baby Mozart being useless and helicopter parenting being harmful. What’s the galaxy brain stuff on raising kids?

Diana: I like Bryan Caplan’s book Selfish Reasons to Have More Kids. But I’ll let Geoffrey speak since he has actually raised kids.

Geoffrey: I have a daughter who is 23, and I also helped raise two stepchildren when they were teenagers.

The most important part of raising kids is to choose a good mate with good genes and then your kids will probably turn out fine. Most people don’t seem to do that very thoughtfully, or to integrate the information from all their extended family to estimate what the kids are likely to be like. It’s important when you meet the parents on Thanksgiving to ask questions and really listen because a lot of things that are heritable won’t necessarily show up in your mate but will show up in a sibling or uncle.

What else? There seems to be a two-generation lag in the skills parents train their kids to do because they were in turn trained by their own parents. There’s not much future orientation. We often talk about how the future is in China and it’s important to learn about China and learn the language…

Diana: We’re talking 15 minutes of Duolingo a day, let’s not get carried away.

Geoffrey: We’re thinking about what the future might be like, technologically and socially. My ambition is to train kids who will, in turn, have kids who will do well on Mars colonies.

Diana: This is literally the first time I’ve heard this. Sending our children to Mars? So they don’t ask us for money?

I saw a study that kids tend to die on the same planet as their biological parents. There’s a perfect correlation, so you shouldn’t worry too much about Geoffrey sending your kids to Mars.

Geoffrey: The third thing is, and I tell my students that, is that there’s too much focus on teaching the things that contribute to status and getting paid and there’s not enough focus on courtship skills. Helping your kids develop things that will be attractive to both friends and the opposite sex, including simple things like singing, drawing, telling jokes and interesting stories — these don’t require much practice but most Gen Z students suck at all those things. They can barely even speak one-on-one, in class or on dates. That seems like an insane handicap to have. Your job as a parent is to help your child hone those courtship skills in addition to skills that are economically relevant.

Are Gen Zers disproportionately worse than previous generations?

Geoffrey: They’re worse. I’ve noticed a gradient over the 30 years I’ve been teaching, kids used to be able to speak in class and to each other more articulately.

Diana: I have a cynical view of children, the same as I do of everything else. I saw a parent bouncing a child up and down to calm them. It’s a very reassuring motion because it’s all-encompassing. It’s very hard for a parent to do that while doing anything else, it signals total attention. Kids like to be rocked in that way precisely because it’s the most costly way to move them.

I don’t know if all the cynicism about children is going to melt away when I become a parent myself. But certainly, when I see my toddler niece have a meltdown, her father reacts in a way that he wouldn’t if he knew she was behaving in a manipulative way. The idea that I have about women training their men, I think similarly about children training their parents.

In terms of galaxy brain parenting, if you never reinforce behaviors like whining, even though some things are genetically determined, you would see less of that behavior. My upbringing gave me some degree of stoicism because my father would just ignore me completely if I was whining or crying. And he was the parent who was most important to me. This idea that you should cater to kids, or that you will cause a child trauma if you let them cry for more than 5 minutes, is I think totally misguided.

Geoffrey: A lot of this is an artifact of family sizes being so small today compared to 100 years ago. My mother was raised in a family of 12 kids. She got by with a remarkably low level of individualized attention from her parents, and she turned out OK. The kids basically attended to each other, the older ones helped take care of the younger ones. And there was no coddling. If one of the teenagers was heartbroken after a breakup, a parent would hear about it a couple of weeks later from one of the older siblings. “Oh, that’s why she was so mopey and didn’t want to practice the violin!”

Kids are very resilient but you wouldn’t know that in a small family.

I have a question about dating. Geoffrey wrote in “Mate” that things like money and status are worthwhile if they ultimately lead to romantic success. If someone is willing to move cities for a job they should certainly be willing to do so for a better mating market, like moving to Manhattan if you’re a young man. And yet, very few people seem to do that.

I’ve spoken to many people who say that romance is as important as their career but they spend 50 hours each week working, 20 hours watching TV, but don’t have a single hour to spare to write a good OKCupid profile. Or people who are better writers than they are good-looking, but are only on Tinder because it’s less work. I know people who won’t commute to the neighboring borough for a date.

How come, if dating is so important, people are so fucking lazy about dating?

Geoffrey: It’s literally tragic, because it’s a lose-lose situation. Good relationships bring happiness to both people. It’s tragic that people suck at mate search in modern societies even though it’s very easy to find awesome partners.

If you have a soulmate model, in which there’s one person out there for you and you’re destined to meet them sooner or later, you’re going to be lazy about trying to meet them. That’s what happens in romantic comedies, where you just end up together in the same elevator that gets stuck and now you’re in love. But that doesn’t work.

Some of the fatalism comes from our ancestors living in tiny societies where there wasn’t really much choice. You probably already knew everyone who might be a potential mate who isn’t your cousin. You knew the tribes over the other hills and who might be in those tribes who is worth raiding or trying to seduce. I think we’re very maladapted to do sustained, rational mate search in these huge mating markets, even though there’s a huge upside to doing that.

Diana: I think you’re talking about rationalists when you’re saying that people don’t spend a lot of effort on mating. I was at a restaurant the other day, one of the waiters chatted up the two women at the table next to me, they left, and then he tried to chat me up even though I just heard him fail with the two women. I already knew his name and his story about the girl he had sex with and the rest of it! And even when I told him that, he was not deterred at all.

I think you’re talking about people who have the self-control to develop rationality and get advanced degrees and good jobs. And perhaps people do those things in order to get the highest status mate, but people get obsessed with all the trappings and lose sight of the final goal.

Regarding Tinder: everyone says that humans are so much more sophisticated than other animals. But on Tinder, all you know is how somebody’s face looks like and how far away they are. You may as well be a raccoon.

Raccoons know each other’s smell, Tinder doesn’t even give you that.

Diana: There’s so little information. Although people say that Tinder provides evidence for physiognomy, that you can tell some things about someone’s personality just from their face.

I do think Tinder is a shame, that people think so little about what they actually want. Although throughout history, if someone lived in proximity to you and was the same age as you, you would often end up together. People with a lot in common ending up together is quite a new thing. Part of why it’s so difficult now is that we know so much about each other, and are trying to find someone we have so much in common with. It’s a new thing. What used to happen is that people would meet, fall in love, and then grow similar to each other over time.

Certainly, the first time I was in love with someone I wanted to do all the things he liked to do, learn about all his interests, etc. At this stage of my life, I probably wouldn’t even have a conversation with him. But back then it was just that I met him and had sex with him. Do you know about voles? Voles have sex and then they just want to be around each other all the time, it’s just primal. I was 16, we had sex, and that was basically it.

What people don’t talk about enough is that as you get older you can assort and think of the characteristics that you like in a potential partner because you don’t fall in love as easily. But this idea that you get less “sticky” each time you have sex with a new person, that you get less attached, it’s a message promoted by Christians. They say you’re like a piece of tape that becomes less sticky each time. But I don’t want to be sticky, thank you very much.

Male and female prairie voles.

This room is full of young and smart people who are interested in how minds work. Would you advise them to go into academic evolutionary psychology? If not, what should they do instead? And if they do enter it, what questions will evolutionary psychology grapple with in the next couple of decades?

Geoffrey: For the love of God, don’t try to become an evolutionary psychologist. The job market sucks. It really sucks. It would be really hard to get a tenure track job. There are maybe 5 positions a year globally in ev psych versus hundreds in social psych or developmental psych. It’s probably not a viable career path.

But, I think reading ev psych can be enormously helpful for understanding yourself, your social life, mating, jobs, consumerism, money, all kinds of things. It’s one of those fields where it can offer a lot of value to the amateur who reads the better books and journals and blogs.

Do you have recommendations?

I have some suggested reading on my site, but that needs to be updated. There’s also everything in my syllabi. The people I know who have taken the time to study this stuff all say it was totally worth it.

Diana: It took me a couple of years to get a job when I was getting into the field, and I was up against 7 or 8 people. The fact I got it had more to do with how I was in the interview; they made all the classic mistakes when they hired me. I think that it can be a very interesting field but what I did was branching out to something else, I did two postdocs. I marketed myself as an evolutionary andhealth psychologist because all the jobs were in health psychology at the time, and that’s how I got my job. If you stick only to evolutionary psychology you’re very unlikely to get a job.

Talk to someone who does the research now, it’s a lot more rigorous. You can’t just come up with an idea, sample 50 undergrads, and publish something. There are really good discoveries to be made but it’s going to be hard for someone who comes along because a lot of those discoveries are things people don’t want to know. We were just talking about ideas like people manipulating each other, selfish gene ideas, these are not the ideas that people enjoy.

So you don’t want competition.

My department actually has an ev psych job coming up this year, for the first time since my colleague Marco del Giudice was hired about 8 years ago.

In terms of the future we are going to see gradual theoretical developments. The more important thing is that ev psych is going to spread globally and enter other countries. It’s pretty strong at the moment in the US, Canada, England, with pockets in Europe, Korea, Japan, and Australia. The real action is going to happen when India and/or China start taking it seriously and develop it.

My superforecaster prediction is that the Chinese government will realize that evolutionary psychology is really important to fix their mating market which is currently really sex-biased. They know that this will result in social instability because tens of millions of young men will not be able to find girlfriends or wives. They know that’s a huge political danger, so they have to do something. It could be free sex bots for everybody, or re-engineering the mating market so that more people are polyamorous, or something. The Chinese government has the capability to do big data analysis of their citizens in a way that we couldn’t possibly do.

In general, I think it’s easier to do large-scale analysis of human behavior if you work for a social media company than if you work in academia. There are 2.4 billion people on Facebook, so the Facebook behavioral sciences team has much richer data than I could ever imagine. But they’re not telling anybody outside what they’re doing, it’s a very siloed kind of science.

The other extreme is that I can run a Twitter poll to my 70k followers and get 4,000 replies in 12 hours. It’s a much bigger scale than I could ever acquire in a lab study, and I don’t need IRB approval for a Twitter poll. I can’t publish the data in a journal, but I and everybody else can learn from it.

There is going to be a bifurcation between official science that gets published in reputable journals, that pre-registered with careful analysis and takes two years to get published. And then fly-by-night informal social media polling on the other hand.

Diana: What I hope to see in the future is rather than universities doing studies with ridiculous ethics boards and so many obstacles, I hope to see more private individuals willing to become patrons of research that falls outside the mainstream. Some of it is happening right now, and it’s something I’m excited about. I hope to see more things privately funded with ethics done in-house according to a set of guidelines — this is going to work much faster and be much more efficient. Doing research through universities can be very onerous.

That’s how we get to CRISPR babies too.

Diana: And what’s wrong with that?

Discuss

### Why artificial optimism?

Новости LessWrong.com - 16 июля, 2019 - 00:41
https://s0.wp.com/i/blank.jpg

### Offering public comment in the Federal rulemaking process

Новости LessWrong.com - 15 июля, 2019 - 23:31
Published on July 15, 2019 8:31 PM UTC

This is a short linkdump of useful resources in the event you are planning to make a public comment on the AI standards. In the main these recommendations are intuitive: it helps to be an expert; identify exactly what document you are referring to; speak directly to your concern; use citations; etc.

How to comment on a rule, from the Center for Effective Government: https://www.foreffectivegov.org/node/4059

How public comment works, from the Public Comment Project:

https://publiccommentproject.org/how-it-works

Tips for submitting effective comments, from Regulations.gov:

How to effectively comment on regulations, from the Brookings Institute:

Examples of public comments, from the Public Comment Project:

https://publiccommentproject.org/comment-examples-index#examples-aqcons

Discuss

### Integrity and accountability are core parts of rationality

Новости LessWrong.com - 15 июля, 2019 - 23:22
Published on July 15, 2019 8:22 PM UTC

Epistemic Status: Pointing at early stage concepts, but with high confidence that something real is here. Hopefully not the final version of this post.

When I started studying rationality and philosophy, I had the perspective that people who were in positions of power and influence should primarily focus on how to make good decisions in general and that we should generally give power to people who have demonstrated a good track record of general rationality. I also thought of power as this mostly unconstrained resource, similar to having money in your bank account, and that we should make sure to primarily allocate power to the people who are good at thinking and making decisions.

That picture has changed a lot over the years. While I think there is still a lot of value in the idea of "philosopher kings", I've made a variety of updates that significantly changed my relationship to allocating power in this way:

• I have come to believe that people's ability to come to correct opinions about important questions is in large part a result of whether their social and monetary incentives reward them when they have accurate models in a specific domain. This means a person can have extremely good opinions in one domain of reality, because they are subject to good incentives, while having highly inaccurate models in a large variety of other domains in which their incentives are not well optimized.
• People's rationality is much more defined by their ability to maneuver themselves into environments in which their external incentives align with their goals, than by their ability to have correct opinions while being subject to incentives they don't endorse. This is a tractable intervention and so the best people will be able to have vastly more accurate beliefs than the average person, but it means that "having accurate beliefs in one domain" doesn't straightforwardly generalize to "will have accurate beliefs in other domains".

One is strongly predictive of the other, and that’s in part due to general thinking skills and broad cognitive ability. But another major piece of the puzzle is the person's ability to build and seek out environments with good incentive structures.
• Everyone is highly irrational in their beliefs about at least some aspects of reality, and positions of power in particular tend to encourage strong incentives that don't tend to be optimally aligned with the truth. This means that highly competent people in positions of power often have less accurate beliefs than much less competent people who are not in positions of power.
• The design of systems that hold people who have power and influence accountable in a way that aligns their interests with both forming accurate beliefs and the interests of humanity at large is a really important problem, and is a major determinant of the overall quality of the decision-making ability of a community. General rationality training helps, but for collective decision making the creation of accountability systems, the tracking of outcome metrics and the design of incentives is at least as big of a factor as the degree to which the individual members of the community are able to come to accurate beliefs on their own.

A lot of these updates have also shaped my thinking while working at CEA, LessWrong and the LTF-Fund over the past 4 years. I've been in various positions of power, and have interacted with many people who had lots of power over the EA and Rationality communities, and I've become a lot more convinced that there is a lot of low-hanging fruit and important experimentation to be done to ensure better levels of accountability and incentive-design for the institutions that guide our community.

I also generally have broadly libertarian intuitions, and a lot of my ideas about how to build functional organizations are based on a more start-up like approach that is favored here in Silicon Valley. Initially these intuitions seemed at conflict with the intuitions for more emphasis on accountability structures, with broken legal systems, ad-hoc legislation, dysfunctional boards and dysfunctional institutions all coming to mind immediately as accountability-systems run wild. I've since then reconciled my thoughts on these topics a good bit.

Integrity

Somewhat surprisingly, "integrity" has not been much discussed as a concept handle on LessWrong. But I've found it to be a pretty valuable virtue to meditate and reflect on.

I think of integrity as a more advanced form of honesty – when I say “integrity” I mean something like “acting in accordance with your stated beliefs.” Where honesty is the commitment to not speak direct falsehoods, integrity is the commitment to speak truths that actually ring true to yourself, not ones that are just abstractly defensible to other people. It is also a commitment to act on the truths that you do believe, and to communicate to others what your true beliefs are.

Integrity can be a double-edged sword. While it is good to judge people by the standards they expressed, it is also a surefire way to make people overly hesitant to update. If you get punished every time you change your mind because your new actions are now incongruent with the principles you explained to others before you changed your mind, then you are likely to stick with your principles for far longer than you would otherwise, even when evidence against your position is mounting.

The great benefit that I experienced from thinking of integrity as a virtue, is that it encourages me to build accurate models of my own mind and motivations. I can only act in line with ethical principles that are actually related to the real motivators of my actions. If I pretend to hold ethical principles that do not correspond to my motivators, then sooner or later my actions will diverge from my principles. I've come to think of a key part of integrity being the art of making accurate predictions about my own actions and communicating those as clearly as possible.

There are two natural ways to ensure that your stated principles are in line with your actions. You either adjust your stated principles until they match up with your actions, or you adjust your behavior to be in line with your stated principles. Both of those can backfire, and both of those can have significant positive effects.

Who Should You Be Accountable To?

In the context of incentive design, I find thinking about integrity valuable because it feels to me like the natural complement to accountability. The purpose of accountability is to ensure that you do what you say you are going to do, and integrity is the corresponding virtue of holding up well under high levels of accountability.

Highlighting accountability as a variable also highlights one of the biggest error modes of accountability and integrity – choosing too broad of an audience to hold yourself accountable to.

There is tradeoff between the size of the group that you are being held accountable by, and the complexity of the ethical principles you can act under. Too large of an audience, and you will be held accountable by the lowest common denominator of your values, which will rarely align well with what you actually think is moral (if you've done any kind of real reflection on moral principles).

Too small or too memetically close of an audience, and you risk not enough people paying attention to what you do, to actually help you notice inconsistencies in your stated beliefs and actions. And, the smaller the group that is holding you accountable is, the smaller your inner circle of trust, which reduces the amount of total resources that can be coordinated under your shared principles.

I think a major mistake that even many well-intentioned organizations make is to try to be held accountable by some vague conception of "the public". As they make public statements, someone in the public will misunderstand them, causing a spiral of less communication, resulting in more misunderstandings, resulting in even less communication, culminating into an organization that is completely opaque about any of its actions and intentions, with the only communication being filtered by a PR department that has little interest in the observers acquiring any beliefs that resemble reality.

I think a generally better setup is to choose a much smaller group of people that you trust to evaluate your actions very closely, and ideally do so in a way that is itself transparent to a broader audience. Common versions of this are auditors, as well as nonprofit boards that try to ensure the integrity of an organization.

This is all part of a broader reflection on trying to create good incentives for myself and the LessWrong team. I will try to follow this up with a post that more concretely summarizes my thoughts on how all of this applies to LessWrong concretely.

In summary:
• One lens to view integrity through is as an advanced form of honesty – “acting in accordance with your stated beliefs.”
• To improve integrity, you can either try to bring your actions in line with your stated beliefs, or your stated beliefs in line with your actions, or reworking both at the same time. These options all have failure modes, but potential benefits.
• People with power sometimes have incentives that systematically warp their ability to form accurate beliefs, and (correspondingly) to act with integrity.
• An important tool for maintaining integrity (in general, and in particular as you gain power) is to carefully think about what social environment and incentive structures you want for yourself.
• Choose carefully who, and how many people, you are accountable to:
• Too many people, and you are limited in the complexity of the beliefs and actions that you can justify.
• Too few people, too similar to you, and you won’t have enough opportunities for people to notice and point out what you’re doing wrong. You may also not end up with a strong enough coalition aligned with your principles to accomplish your goals.

[This post was originally posted on my shortform feed]

Discuss

### Jeff Hawkins on neuromorphic AGI within 20 years

Новости LessWrong.com - 15 июля, 2019 - 22:16
Published on July 15, 2019 7:16 PM UTC

I just listened to AI podcast: Jeff Hawkins on the Thousand Brain Theory of Intelligence, and read some of the related papers. Jeff Hawkins is a theoretical neuroscientist; you may have heard of his 2004 book On Intelligence. Earlier, he had an illustrious career in EECS, including inventing the Palm Pilot. He now runs the company Numenta, which is dedicated to understanding how the human brain works (especially the neocortex), and using that knowledge to develop bio-inspired AI algorithms.

In no particular order, here are some highlights and commentary from the podcast and associated papers.

Every part of the neocortex is running the same algorithm

The neocortex is the outermost and most evolutionarily-recent layer of the mammalian brain. In humans, it is about the size and shape of a dinner napkin (maybe 1500cm²×3mm), and constitutes 75% of the entire brain. Jeff wants us to think of it like 150,000 side-by-side "cortical columns", each of which is a little 1mm²×3mm tube, although I don't think we're supposed to the "column" thing too literally (there's no sharp demarcation between neighboring columns).

When you look at a diagram of the brain, the neocortex has loads of different parts that do different things—motor, sensory, visual, language, cognition, planning, and more. But Jeff says that all 150,000 of these cortical columns are virtually identical! Not only do they each have the same types of neurons, but they're laid out into the same configuration and wiring and larger-scale structures. In other words, there seems to be "general-purpose neocortical tissue", and if you dump visual information into it, it does visual processing, and if you connect it to motor control pathways, it does motor control, etc. He said that this theory originated with Vernon Mountcastle in the 1970s, and is now widely (but not universally) accepted in neuroscience. The theory is supported both by examining different parts of the brain under the microscope, and also by experiments, e.g. the fact that congenitally blind people can use their visual cortex for non-visual things, and conversely he mentioned in passing some old experiment where a scientist attached the optic nerve of a lemur to a different part of the cortex and it was able to see (or something like that).

Anyway, if you accept that premise, then there is one type of computation that the neocortex does, and if we can figure it out, we'll understand everything from how the brain does visual processing to how Einstein's brain invented General Relativity.

To me, cortical uniformity seems slightly at odds with the wide variety of instincts we have, like intuitive physics, intuitive biology, language, and so on. Are those not implemented in the neocortex? Are they implemented as connections between (rather than within) cortical columns? Or what? This didn't come up in the podcast.

Grid cells and displacement cells Background: Grid cells for maps in the hippocampus

Grid cells, discovered in 2005, help animals build mental maps of physical spaces. (Grid cells are just one piece of a complicated machinery, along with "place cells" and other things, more on which shortly.) Grid cells are not traditionally associated with the neocortex, but rather the entorhinal cortex and hippocampus. But Jeff says that there's some experimental evidence that they're also in the neocortex, and proposes that this is very important.

What are grid cells? Numenta has an educational video here. Here's my oversimplified 1D toy example (the modules can also be 2D). I have a cortical column with three "grid cell modules". One module consists of 9 neurons, one has 10 neurons, and the third has 11. As I stand in a certain position in a room, one neuron from each of the three modules is active - let's say the active neurons right now are (x1 mod 9), (x2 mod 10), and (x3 mod 11) for some integers x1,x2,x3. When I take a step rightward, x1,x2,x3 are each incremented by 1; when I take a step leftward, they're each decremented by 1. The three modules together can thus keep track of 990 unique spatial positions (cf. Chinese Remainder Theorem).

With enough grid cell modules of incommensurate size, scale-factor, and (in 2D) rotation, the number of unique representable positions becomes massive, and there is room to have lots of entirely different spaces (each with their own independent reference frame) stored this way without worrying about accidental collisions.

So you enter a new room. Your brain starts by picking a point in the room and assigns it a random x1,x2,x3 (in my toy 1D example), and then stores all the other locations in the room in reference to that. Then you enter a hallway. As you turn your attention to this new space, you pick a new random x′1,x′2,x′3 and build your new hallway spatial map around there. So far so good, but there's a missing ingredient: the transformation from the room map to the hallway map, especially in their areas of overlap. How does that work? Jeff proposes (in this paper) that there exist what he calls "displacement cells", which (if I understand it correctly) literally implement modular arithmetic for the grid cell neurons in each grid cell module. So⁠—still in the 1D toy example⁠—the relation between the room map and the hall map might be represented by three displacement cell neurons δ1,δ2,δ3 (one for each of the three grid cell modules), and the neurons are wired up such that the brain can go back and forth between the activations {(x1 mod 9),(x2 mod 10),(x3 mod 11)}↔↔{((x1+δ1) mod 9),((x2+δ2) mod 10),((x3+δ3) mod 11)}.

So if grid cell #2 is active, and then displacement cell #5 turns on, it should activate grid cell #7=5+2. It's kinda funny, but why not? We just put in a bunch of synapses that hardcode each entry of an addition table⁠—and not even a particularly large one.

(Overall, all the stuff about the detailed mechanisms of grid cells and displacement cells comes across to me as "Ingenious workaround for the limitations of biology", not "Good idea that AI might want to copy", but maybe I'm missing something.)

New idea: Grid cells for "maps" of objects and concepts in the neocortex

Anyway, Jeff theorizes that this grid cell machinery is not only used for navigating real spaces in the hippocampus but also navigating concept spaces in the neocortex.

Example #1: A coffee cup. We have a mental map of a coffee cup, and you can move around in that mental space by incrementing and decrementing the xi (in my 1D toy example).

Example #2: A coffee mug with a picture on it. Now, we have a mental map of the coffee mug, and a separate mental map of the picture, and then a set of displacement cells describe where the picture is in relation to the coffee cup. (This also includes relative rotation and scale, which I guess are also part of this grid cell + displacement cell machinery somehow, but he says he hasn't worked out all the details.)

Example #3: A stapler, where the two halves move with respect to each other. This motion can be described by a sequence of displacement cells ... and conveniently, neurons are excellent at learning temporal sequences (see below).

Example #4: Logarithms. Jeff thinks we have a reference frame for everything! Every word, every idea, every concept, everything you know has its own reference frame, in at least one of your cortical columns and probably thousands of them. Then displacement cells can encode the mathematical transformations of logarithms, and the relations between logarithms and other concepts, or something like that. I tried to sketch out an example of what he might be getting at in the next section below. Still, I found that his discussion of abstract cognition was a bit sketchier and more confusing than other things he talked about. My impression is that this is an aspect of the theory that he's still working on.

"Thousand brains" theory

(See also Numenta educational video.) His idea here is that every one of the 150,000 "cortical columns" in the brain (see above) has the whole machinery with grid cells and displacement cells, reference frames for gazillions of different objects and concepts, and so on.

A cortical column that gets input from the tip of the finger is storing information and making predictions about what the tip of the finger will feel as it moves around the coffee cup. A cortical column in the visual cortex is storing information and making predictions about what it will see in its model of the coffee cup. And so on. If you reach into a box, and touch it with four fingers, each of those fingers is trying to fit its own data into its own model to learn what the object is, and there's a "voting" mechanism that allows them to reach agreement on what it is.

So I guess if you're doing a math problem with a logarithm, and you're visually imagining the word "log" floating to the other side of the equation and turning into an "exp", then there's a cortical column in your visual cortex that "knows" (temporal sequence memory) how this particular mathematical transformation works. Maybe the other cortical columns don't "know" that that transformation is possible, but can find out the result via the voting mechanism.

Or maybe you're doing the same math problem, but instead of visualizing the transformation, instead you recite to yourself the poem: "Inverse of log is exp". Well, then this knowledge is encoded as the temporal sequence memory in some cortical column of your auditory cortex.

There's a homunculus-esque intuition that all these hundreds of thousands of models need to be brought together into one unified world model. Neuroscientists calls this the "sensor fusion" problem. Jeff denies the whole premise. Thousands of different incomplete world models, plus a voting mechanism, is all you need; there is no unified world model.

Is the separate world model for each cortical column an "Ingenious workaround for the limitations of biology" or a "Good idea that AI should copy"? On the one hand, clearly there's some map between the concepts in different cortical columns, so that voting can work. That suggests that we can improve on biology by having one unified world model, but with many different coordinate systems and types of sensory prediction associated with each entry. On the other hand, maybe the map between entries of different columns' world models is not a nice one-to-one map, but rather some fuzzy many-to-many map. Then unifying it into a single ontology might be fundamentally impossible (except trivially, as a disjoint union). I'm not sure. I guess I should look up how the voting mechanism is supposed to work.

Human-level AI, timelines, and existential risk

Jeff's goal is to "understand intelligence" and then use it to build intelligent machines. He is confident that this is possible, and that the machines can be dramatically smarter than humans (e.g. thinking faster, more memory, better at physics and math, etc.). Jeff thinks the hard part is done—he has the right framework for understanding cortical algorithms, even if there are still some details to be filled in. Thus, Jeff believes that, if he succeeds at proselytizing his understanding of brain algorithms to the AI community (which is why he was doing that podcast), then we should be able to make machines with human-like intelligence in less than 20 years.

Near the end of the podcast, Jeff emphatically denounced the idea of AI existential risk, or more generally that there was any reason to second-guess his mission of getting beyond-human-level intelligence as soon as possible. However, he appears to be profoundly misinformed about both what the arguments are for existential risk and who is making them—almost as if he learned about the topic by reading Steven Pinker or something. Ditto for Lex, the podcast host.

Differences between actual neurons and artifical neural networks (ANNs) Non-proximal synapses and recognizing time-based patterns

He brought up his paper Why do neurons have thousands of synapses?. Neurons have anywhere from 5 to 30,000 synapses. There are two types. The synapses near the cell body (perhaps a few hundred) can cause the neuron to fire, and these are most similar to the connections in ANNs. The other 95% are way out on a dendrite (neuron branch), too far from the neuron body to make it fire, even if all 95% were activated at once! Instead, what happens is if you have 10-40 of these synapses that all activate at the same time and are all very close to each other on the dendrite, it creates a "dendritic spike" that goes to the cell body and raises the voltage a little bit, but not enough to make the cell fire. And then the voltage goes back down shortly thereafter. What good is that? If the neuron is triggered to fire (due to the first type of synapses, the ones near the cell body), and has already been prepared by a dendritic spike, then it fires slightly sooner, which matters because there are fast inhibitory processes, such that if a neuron fires slightly before its neighbors, it can prevent those neighbors from firing at all.

So, there are dozens to hundreds of different patterns that the neuron can recognize—one for each close-together group of synapses on a dendrite—each of which can cause a dendritic spike. This allows networks of neurons to do sophisticated temporal predictions, he says: "Real neurons in the brain are time-based prediction engines, and there's no concept of this at all" in ANNs; "I don't think you can build intelligence without them".

Another nice thing about this is that a neuron can learn a new pattern by forming a new cluster of synapses out on some dendrite, and it won't alter the neuron's other behavior—i.e., it's an OR gate, so when that particular pattern is not occurring, the neuron behaves exactly as before.

Binary weights, sparse representations

Another difference: "synapses are very unreliable"; you can't even assign one digit of precision to their connection strength. You have to think of it as almost binary. By contrast, I think most ANN weights are stored with at least ~2 and more often 7 decimal digits of precision.

Related to this, "the brain works on sparse patterns". He mentioned his paper How do neurons operate on sparse distributed representations? A mathematical theory of sparsity, neurons and active dendrites. He came back to this a couple times. Apparently in the brain, at any given moment, ~2% of neurons are firing. So imagine a little subpopulation of 10,000 neurons, and you're trying to represent something with a population code of sets of 200 of these neurons. First, there's an enormous space of possibilities (10424). Second, if you pick two random sets-of-200, their overlap is almost always just a few. Even if you pick millions of sets, there won't be any pair that significantly overlaps. Therefore a neuron can "listen" for, say, 15 of the 200 neurons comprising X, and if those 15 all fire at once, that must have been X. The low overlap between different sets also gives the system robustness, for example to neuron death. Based on these ideas, they recently published this paper advocating for sparseness in image classifier networks, which sounds to me like they're reinventing neural network pruning, but maybe it's slightly different, or at least better motivated.

Learning and synaptogenesis

According to Jeff, the brain does not learn by changing the strength of synapses, but rather by forming new synapses (synaptogenesis). Synaptogenesis takes about an hour. How does short-term memory work faster than that? There's something called "silent synapses", which are synapses that don't release neurotransmitters. Jeff's (unproven) theory is that short-term memory entails the conversion of silent synapses into active synapses, and that this occurs near-instantaneously.

Vision processing

His most recent paper has this image of image processing in the visual cortex:

As I understand it, the idea is that every part of the field of view is trying to fit what it's looking at into its own world model. In other words, when you look at a cup, you shouldn't be thinking that the left, center, and right parts of the field-of-view are combined together and then the whole thing is recognized as a coffee cup, but rather that the left part of the field-of-view figures out that it's looking at the left side of the coffee cup, the center part of the field-of-view figures out that it's looking at the center of the coffee cup, and the right part of the field-of-view figures out that it's looking at the right side of the coffee cup. This process is facilitated by information exchange between different parts of the field-of-view, as well as integrating the information that a single cortical column sees over time as the eye or coffee cup moves. As evidence, they note that there are loads of connections in the visual cortex that are non-hierarchical (green arrows). Meanwhile, the different visual areas (V1, V2, etc.) are supposed to operate on different spatial scales, such that a faraway cup of coffee (taking up a tiny section of your field-of-view) might be recognized mainly in V1, while a close-up cup of coffee (taking up a larger chunk of your field-of-view) might be recognized mainly in V4, or something like that.

Maybe this has some profound implications for building CNN image classifiers, but I haven't figured out what exactly they would be, other than "Maybe try putting in a bunch of recurrent, non-hierarchical, and/or feedback connections?"

My conclusions for AGI safety

Jeff's proud pursuit of superintelligence-as-fast-as-possible is a nice reminder that, despite the mainstreaming of AGI safety over the past few years, there's still a lot more advocacy and outreach work to be done. Again, I'm concerned not so much about the fact that he disagrees with arguments for AGI existential risks, but rather that he (apparently) has never even heard the arguments for AGI existential risks, at least not from any source capable of explaining them correctly.

As for paths and timelines: I'm not in a great position to judge whether Jeff is on the right track, and there are way too many people who claim to understand the secrets of the brain for me to put a lot of weight on any one of them being profoundly correct. Still, I was pretty impressed, and I'm updating slightly in favor of neuromorphic AGI happening soon, particularly because of his claim that the whole neocortex is more-or-less cytoarchitecturally uniform.

Finally, maybe the most useful thing I got out of this is fleshing out my thinking about what an AGI's world model might look like.

Jeff is proposing that our human brain's world models are ridiculously profligate in the number of primitive entries included. Our world models don't just have one entry for "shirt", but rather separate entries for wet shirt, folded shirt, shirt-on-ironing-board, shirt-on-floor, shirt-on-our-body, shirt-on-someone-else's-body, etc. etc. etc. After all, each of those things is associated with a different suite of sensory predictions! In fact, it's even more profligate than that: Really, there might be an entry for "shirt on the floor, as it feels to the center part of my left insole when I step on it", and an entry for "my yellow T-shirt on the floor, as it appears to the rod cells in my right eye's upper peripheral vision". Likewise, instead of one entry for the word "shirt", there are thousands of them in the various columns of the auditory cortex (for the spoken word), and thousands more in the columns of the visual cortex (for the written word). To the extent that there's any generic abstract concept of "shirt" in the human brain, it would probably be some meta-level web of learned connections and associations and transformations between all these different entries.

If we build an AI which, like the human brain, has literally trillions of primitive elements in its world model, it seems hopeless to try to peer inside and interpret what it's thinking. But maybe it's not so bad? Let's say some part of cortical column #127360 has 2000 active neurons at some moment. We can break that down into 10 simultaneous active concepts (implemented as sparse population codes of 200 neurons each), and then for each of those 10, we can look back at the record of what was going on the first time that code ever appeared. We can look at the connections between that code and columns of the language center, and write down all those words. We can look at the connections between that code and columns of the visual cortex, and display all those images. Probably we can figure out more-or-less what that code is referring to, right? But it might take 1000 person-years to interpret one second of thought by a human-brain-like AGI! (...Unless we have access to an army of AI helpers, says the disembodied voice of Paul Christiano....) Also, some entries of the world model might be just plain illegible despite our best efforts, e.g. the various neural codes active in Ed Witten's brain when he thinks about theoretical physics.

Discuss

### Commentary On "The Abolition of Man"

Новости LessWrong.com - 15 июля, 2019 - 21:56
Published on July 15, 2019 6:56 PM UTC

C.S. Lewis wrote a short book attacking moral subjectivism in education; it's available online here as a pdf, here as raw text, and here as a series of videos [1 2 3], and I think probably worth reading in full (at 50 pages or ~100 minutes of video at 1x speed). This post is mostly me rambling about what I saw as the central point, especially connected to individual development and community health, by quoting sections and then reacting to them.

The book begins with a reaction to a grammar textbook (written in 1939) whose lessons are also philosophical; Lewis doesn't object to the bait-and-switch (outside of one paragraph) so much as the content and quality of the philosophy. (One suspects Lewis wouldn't object to the Copybook Headings referenced by Kipling, even tho that mixes writing lessons and philosophy.)

Until quite modern times all teachers and even all men believed the universe to be such that certain emotional reactions on our part could be either congruous or incongruous to it--believed, in fact, that objects did not merely receive, but could merit, our approval or disapproval, our reverence or our contempt.

First, let's get the obvious objections out of the way: the claim of universality is probably false. Even supposing it were true, then the underlying change seems worth investigating. Naive belief that one's map is objective reality disintegrates on contact with different maps and after noticing surprising divergences between one's predictions and observations; one can imagine this happening in the moral realm as well as the physical one. But presumably we should just ignore this as standard "the contemporary world is fallen and bad" framing instead of an actual historical claim.

The more interesting claim here is the question of whether or not there can or should be a question of merit, distinct from a question of flavor or fact. A previous taxonomy I've liked a lot (that I was mostly introduced to by Sapiens) is the split between objective (determined by reality), subjective (determined by the person in question), and intersubjective (determined by some group process); the rules of a game are not just 'my personal whims' and are also not 'scientific' in the sense that any outside observer would be able to determine it themselves. Without access to human civilization; aliens would figure out the same physics, and they might play something like chess, but they likely won't play chess. Nevertheless, concepts like chess are an important component of your epistemology and there is such a thing as a 'legal move' or 'illegal move.'

But what is common to [religious traditions] is something we cannot neglect. It is the doctrine of objective value, the belief that certain attitudes are really true, and other really false, to the kind of thing the universe is and the kind of things we are. Those who know the Tao can hold that to call children delightful or old men venerable is not simply to record a psychological fact about our own parental or filial emotions at the moment, but to recognize a quality which demands a certain response from us whether we make it or not.”

Lewis is trying to go a step further; in my framing, there's a thing about the 'game that is society' that involves 'playing with reality' in a way that makes it something a little more objective than the 'intersubjective.' It's not just that everyone jointly decided that old people are venerable and thus the fashion is to venerate them; it's that somehow venerating old people is congruous with the Tao and not venerating them isn't, and so getting that question wrong is worse on some dimension than just playing chess by the wrong rules. Play chess by the wrong rules and people will throw you out of the chess club; play society by the wrong rules and your society collapses or misery abounds. Lewis uses 'the Tao' to refer to both 'the underlying territory as distinct from the map' and 'the sort of human behavior congruous with the territory', in a way that seems connected to this sense of 'the universe as participant in the game that is society.'

Note that he says "true to the kind of thing the universe is and the kind of things we are", as opposed to simply "true." This seems consistent with 'morality as the product of game theory', and a sort of subjectivism that allows for different environments to have different moralities, or different professions to have different ethics; the Tao of the soldier may be distinct from the Tao of the doctor, and the Tao of the Inuit different from the Tao of the Swahili. It reminds me of the claim that Probability is Subjectively Objective; if one is a soldier, the 'right way to be' is different than if one is a doctor, but there is still a meaningful sense in which there is only 'one right way to be' that is not destroyed by that variation. [Imagine a function from 'broad situation' to 'proper behavior'; this function can vary as you change the input while still being a deterministic function.]

If they embark on this course the difference between the old and the new education will be an important one. Where the old initiated, the new merely ‘conditions’. The old dealt with its pupils as grown birds deal with young birds when they teach them to fly; the new deals with them more as the poultry-keeper deals with young birds--making them thus or thus for purposes of which the birds know nothing. In a word, the old was a kind of propagation--men transmitting manhood to men; the new is merely propaganda.

The contrast between 'initiation' and 'conditioning' stuck out to me. One way you could get such a split is a separation between Educators and Students where most students will not become educators, whereas most boy-children become men. When I try to figure out what the difference between religions and cults are, especially when it comes to things like the rationality community, I keep thinking about this sense of "explorers trying to create more explorers", and how it differs from "carnies trying to use marks", and somehow it seems connected to feedback loops. The man trying to make the next generation into men relates to the next generation differently from how the carnie trying to extract money from marks relates to those marks. Not only does the former involve identification with the audience (where the latter recoils from that), the former is trying to get the audience to understand the whole process (so that they too, in their time, can perform it), whereas the latter is trying to get the audience to misunderstand the whole process (so that they will come back and be fleeced again).

To the extent that the High Modernist or Reformer or Rationalist sees the outside as a thing to be optimized, as opposed to part of a system that needs to support further optimization, it seems like there's some deep short-sightedness and disconnection from the Tao. To the extent that some profession sees the outside world as something to be profited from, as opposed to a body in which they are an organ, we should expect the society to be sick in some way.

Let us suppose for a moment that the harder virtues could really be theoretically justified with no appeal to objective value. It still remains true that no justification of virtue will enable a man to be virtuous. Without the aid of trained emotions the intellect is powerless against the animal organism. … The head rules the belly through the chest--the seat, as Alanus tells us, of Magnanimity, of emotions organized by trained habit into stable sentiments. The Chest-Mananimity-Sentiment--these are the indispensable liaison officers between cerebral man and visceral man. It may even be said that it is by this middle element that man is man; for by his intellect he is a mere spirit and by his appetite mere animal.The operation of The Green Book and its kind is to produce what may be called Men without Chests. It is an outrage that they should be commonly spoken of as Intellectuals. This gives them the chance to say that he who attacks them attacks Intelligence.

This reminded me of Bayesians vs. Barbarians, with a new dimension added; it is not that the Barbarians gain from having less in their head, it is that the Bayesians lost because they forgot to develop their chests. When I was younger, I read through The Fountainhead and Atlas Shrugged and was confused by the educational strategy; here were these staunchly moral characters, as evidenced by their disgust at taking immoral actions that would benefit them, but the source of their morality seemed unspoken and unjustified. This felt like a serious contrast to what I observed at my local church, where people put in serious amounts of effort to become slightly more aligned with their reasoned values. It looked like all that was assumed unnecessary; one simply had to paint the picture of correctness and it would be followed by the righteous without any exercise or training.

Another Eliezer reference is Feeling Rational, which points at the congruity property of emotions, but only with regards to factual truth; if you're afraid about an iron being hot and it's cold, you're making a mistake, and if you're calm about an iron being cold and it's hot, you're making a mistake. But that seems to miss the intersubjective angle; in some contexts, reacting to criticism with defensiveness is inappropriate and reacting to criticism with curiosity is appropriate, and some large part of 'training human rationality' is inculcating the right emotional responses in oneself. A dojo isn't just about transfer of technique, but also about transfer of attitude.

Discuss

### Overcoming Akrasia/Procrastination - Volunteers Wanted

Новости LessWrong.com - 15 июля, 2019 - 21:29
Published on July 15, 2019 6:29 PM UTC

I've got some new material I'm working on a related to overcoming procrastination and akrasia. It breaks down the skill of focus/willpower into seven "mental habits" that when stacked, are what you need to overcome procrastination. I wrote a bit more about what the habits are on my Shortform Feed.

I'd love to test teaching the material with a few cohorts before I finalize it into a course. The commitment would be 30 minutes a day for 7 days, free for the first few cohorts. The sessions are about 1/3 teaching, 2/3 practice of the skills. If anyone here is interested let me know.

Discuss

### Рабочая группа по байесовской статистике

События в Кочерге - 15 июля, 2019 - 19:30
Читаем учебник Statistical Rethinking, Chapman&Hall (теория и задачи, английский язык и язык R), поглядываем в The Book of Why, Judea Pearl и моделируем реальные кейсы, используя всё, что успели освоить.

### How Should We Critique Research?

Новости LessWrong.com - 15 июля, 2019 - 01:51
Published on July 14, 2019 10:51 PM UTC

Discuss

### Insights from Linear Algebra Done Right

Новости LessWrong.com - 13 июля, 2019 - 21:24
Published on July 13, 2019 6:24 PM UTC

This book has previously been discussed by Nate Soares and TurnTrout. In this... review? report? detailed journal entry? ... I will focus on the things which stood out to me. (Definitely not meant to be understandable for anyone unfamiliar with the field.) The book has ten chapters; I did most of the exercises in chapters 6 to 10. I got through that half of the book in about 3 weeks, which was nice since the previous (harder and longer) textbook on topology I worked through took me about a year.

I've previously taken two introductory courses to liner algebra and came out thinking of the field as large and very unstructured, with lots of concepts and theorems floating around disconnectedly. What is a normal matrix? I might or might not have remembered the definition, but I certainly had no idea what it is good for. Three weeks of intense study with this book has improved my understanding dramatically. Now, the field seems structured and pretty intuitive (and I certainly won't forget what a normal operator is). It is truly hard to overstate how good of a job this book does teaching the subject, compared to what I've seen before. It's probably the best textbook on anything I've read so far.

Chapter 1 introduces complex numbers and vectors spaces.

Like most fields in math, Linear Algebra ultimately cares about studying functions, and once again it only cares about a tiny subset of all possible functions. In this case it is linear maps, which are the focus of Chapter 3. These are functions T:V→W such that T(x+y)=T(x)+T(y) and T(αx)=αT(x) for all x,y∈V and α∈F, where V and W are vector spaces over some field F (in this book, always R or C).

Brief comparison to topology: The word "continuity" is only mentioned in the final chapter of this book, but every linear map can be shown to be continuous if the topologies on both vector space are given by an inner product. Actually, every inner product induces a norm, every norm induces a metric, any metric induces a topology, but none of the reverse steps are true. Put differently, metric topologies are particularly nice topologies, metrics from norms are particularly nice metrics, norms from inner-products are particularly nice norms, and inner products are the concept studied in chapters 6 and 7. So Linear Algebra only even bothers with the the super nicely behaved stuff.

I'll now detail my biggest confusion with LA before reading this book (this is something I've even brought up explicitly, but without getting a satisfying answer): if T:R2→R3 is a linear map, then it is actually possible to determine its behavior entirely by fixing bases (x,y) of R2 and (v,w,z) of R3 and storing six numbers

⎡⎢⎣a1,1a1,2a2,1a2,2a3,1a3,2⎤⎥⎦

which tell us that T(x)=a1,1v+a2,1w+a3,1z and T(y)=a1,2v+a2,2w+a3,2z. (This determines the behavior of T on all vectors because of linearity of T and the fact that (x,y) and (v,w,z) are bases). The matrix is dependent on the two bases chosen here (given these bases, the set of all linear maps from R2 to R3 is bijective to that of 3-by-2 matrices).

Linear maps aren't relative to bases. Vectors were introduced as elements of Rn, so not relative to bases either, but then they were introduced again in "coordinate form", where we write (a,b)B (B is our basis) to mean "a times the first basis vector plus b times the second," and suddenly a dependence to our basis has been created. My confusion was then: is the B in the index just a reminder that (a,b) is understood to be relative to that basis, or is it a function that maps the vector (a,b) to the vector (ax+by)? In the former case, how should one differentiate between coordinate vectors and non-coordinate vectors? (We've used the same notation.) And how do we specify the vectors in a basis? If I write (1,0) and that means "1 times the first basis vector, 0 times the second" while trying to specify my basis, then clearly that's no good. And the latter case is even worse: now matrix-with-vector-multiplication is ill-defined. Consider the standard basis B=((1,0),(0,1)) on R2 and the basis B′=((0,1),(1,0)) with both vectors reversed. Then we have

(1000)(10)B=(1000)(01)B′=(00)B′=(00)B

But, clearly

(1000)(10)B=(10)B

So we have a flat out contradiction. And it gets worse yet if A is a matrix and one defines a linear map f:V→V by f:v↦Av, which was actually done in one of my past lectures.

So that was the problem. In contrast, here's what the book does. Vectors are elements in Rn, not relative to bases. Maps are functions Rn→Rm, likewise not relative to bases. Matrices are a bunch of numbers put into an array, and one defines the matrix of a linear map T, denoted M(T,B,B′), in the way I've described above (so relative to two bases). Most importantly, there is no matrix-vector multiplication. There is matrix-matrix multiplication, which is of course fine because both matrices are relative to bases if they come from linear maps. To "apply" a matrix to a vector, one defines, given a vector v and a basis B, the matrix of a vector M(v,B) as the unique n-by-1 matrix

⎡⎢ ⎢⎣a1⋮an⎤⎥ ⎥⎦

whose entries are the scalars needed to write v as a sum of basis vectors of B. Then one proves that M(T,B,B′)⋅M(v,B)=M(T(v),B′). Note the square brackets: the book uses () brackets for lists (lists are the same as tupels, i.e. elements of Rnand vectors are lists), and [] brackets for matrices only, which is a solid convention. Does this approach resolve this problem to full satisfaction and without introducing any new difficulties? I think so.

Chapter 4 introduces polynomials. It doesn't have much LA in it, but understanding polynomials well is important for LA, because polynomials can be applied to operators (those are linear maps of the form T→T, i.e. from a vector space into itself). This is so because, given T:Fn→Fn, we can define (cT)(x)=c⋅T(x) if c∈F and T2(x)=T(T(x)) and T0=I (the identity map).

The other special thing about the book – and the justification the author gives for calling it "Linear Algebra Done Right" – is that determinants aren't introduced until the final chapter. This is very unusual: most courses, such as the lectures I took, frequently use determinants in proofs. But all of these proofs can be done without them. And they tend to be fairly simple, too; only a couple are longer than a page.

Doing this, one finds that there are striking parallels between properties about polynomials and properties about linear vector spaces, which seems a lot better for developing a useful intuition than what one gets by doing determinant-based proofs. The central theorem about operators introduced in Chapter 5, for example, states that every operator on a finite-dimensional complex vector space has an eigenvalue (that is, a scalar a∈F such that there exists a nonzero vector v such that Tv=av). This can be proven by essentially applying the fundamental theorem of algebra. One chooses any nonzero vector x∈V and constructs the list (x,Tx,T2x,...,Tnx), where n=dim(V). These vectors cannot be linearly independent because they are n+1 many, so there exist scalars αj such that ∑nj=0αjTjx=0. Now (or rather, after doing some preliminary work) the fundamental theorem of algebra kicks in, and we can reformulate this as (T−β1I)⋯(T−βtI)x=0 where the βj are the roots of the polynomial p(X)=∑nj=0αjXj. Since the RHS equals 0, one of the (T−βjI) isn't injective, and therefore T has the respective βj as an eigenvalue. This is done properly in the book in relatively little space. It also shows why the same theorem fails to hold on real vector spaces: the reformulation of the polynomial to(T−β1I)⋯(T−βtI)x=0 might no longer be possible.

Chapter 6 introduces inner products. These are the generalized formalization underlying the concept of angle between vectors. There is a famous inequality, called the Cauchy-Schwartz inequality, which states that |⟨x,y⟩|≤||x||⋅||y||. The book gives a very nice proof for this that I find easier to remember than the proofs I've seen previously. It is based on the orthogonal decomposition of a vector, where if x and y are any nonzero vectors, then x equals y+az for some vector z and some scalar a such that y and z are orthogonal to each other. I hadn't heard of this before and it's quite useful. If one remembers the formula for the orthogonal decomposition, plugs it into the inner product and does the obvious things, the Cauchy-Schwartz inequality pops out at the other end.

The absolute most nicest things ever studied anywhere are orthonormal bases. These are lists of vectors (e1,...,en) such that a) they are all linearly independent, b) they span the entire space (follows from a) if there are dim(V) many), c) they are all orthogonal to each other and d) they all have length 1. Because this is LA, orthonormal bases always exist and there is even an explicit algorithm to construct one from any existing non-orthonormal basis. Something really cool that the book does with this (which also requires a neat theorem about minimal distance of a vector to a subspace) is to construct the polynomial of degree at most 5 that is the most similar to the sin function on the interval [−π,π] (based on some reasonable definition of similarity involving integrals). I say this is really cool because I found it to be a surprisingly strong result, and a surprising kind of result coming out of linear algebra, given that there is really nothing linear about the sin function. On the interval [−π,π], this approximation looks indistinguishable to sin to the normal eye.

Back to orthonormal bases: what their existence means is that, whenever one tries to prove anything involving inner product spaces, one can simply say something like, "let {e1,...,en} be an orthonormal basis for V and let {e′1,...,e′m} be an orthonormal basis for W", and then proceed to reduce whatever argument one needs to the basis elements. This feels very powerful, even if arbitrarily difficult problems exist regardless.

With Chapter 7, we're getting to the big classification theorems. LA wants to study linear maps, and in this chapter it classifies all of the linear maps such that there exists an orthonormal basis relative to which they have diagonal form. This is the aforementioned spectral theorem. These are the normal operators on complex vector spaces, and the self-adjoint operators (which is a strictly stronger property) on real vector spaces. This highlights another recurring pattern: there are often two versions of each result, one super nice one about complex vector spaces, and a less nice, more cumbersome one about real vector spaces. This makes a lot of sense given that C is far more nicely behaved than R (this is kind of why we defined complex numbers in the first place). More specifically, it usually comes down to the fact that every polynomial of degree ≥1 has a root in C, but not necessarily in R, just like it did in the aforementioned result on the existence of eigenvalues. The book also draws an interesting parallel between self-adjoint operators and real numbers.

In Chapters 8 and 9, we return to the study of eigenvalues. Basically, the problem is as follows: given an operator T:V→V, one would like to find a decomposition of V into smaller subspaces V1,⋯,Vn such that a) each vector v∈V can be uniquely written as a weighted sum of the vi, i.e. v=∑nk=1αkvk with vk∈Vk and b) the operator T maps each subspace Vk into itself, i.e. T:Vk→Vk. If this is achieved, then the behavior of T is determined entirely by the behavior of the T|Vk (i.e. T restricted to Vk); and this is great because the T|Vk are much easier to handle than T. In fact, if the Vk are one-dimensional vector spaces (i.e. just straight lines, if interpreted geometrically) then T|Vk just multiplies each vector with a scalar. In other words, T|Vkvk=Tvk=avk. This is of course just the equation that says that a is an eigenvalues of T with nonzero eigenvector vk, and so that's the reason why eigenvalues are of interest. Conversely, given an eigenvalue a with eigenvector vk, the corresponding one-dimensional subspace is simply {avk|a∈F}.

As stated before, every operator on a complex vector spaces has some eigenvalue. But does it have n different eigenvalues, so that V=V1⊕⋯⊕Vn? (The symbol ⊕ means "direct sum".) In general, the answer is yes – by which I mean that, if one generates an operator randomly (by sampling the coefficients of its matrix relative to the standard basis randomly out of some large finite set), then as one increases the size of that set (for example by switching from 32 bit to 64 bit to 128 bit floating point arithmetic and so on), the probability of generating an operator for which this doesn't work converges to 0. And I believe it is also possible to define a non-discrete probability space such as [0,1]n2, give it the uniform distribution, and then prove that the set of operators for which this doesn't work has probability mass 0.

So for "general" operators, this is always true, but there do exist specific operators for which it isn't. One such example is the operator T whose matrix wrt the standard basis is [1101]. This operator has the eigenvalue 1 but the only eigenvectors are of the form (0,x). Thus there exists one one-dimensional subspace that is invariant under T, namely {(0,x)|x∈F}, but not a second one, so there isn't any decomposition of V=C2 into two invariant subspaces.

To remedy this, Chapter 8 introduces generalized eigenvectors. Observe that the property Tx=ax can also be written as (T−aI)x=0 (where I is the identity map) or as (T|S−aI|S)=0, where S={ax|a∈F}, i.e. S is the one-dimensional vector space containing x. The equation above then says that, if we take T, restrict it to S and subtract a times the identity operator, we get the zero operator (so the 0 on the right denotes the null map Z:S→{0}). This reformulation leads to the non-obvious definition of a generalized eigenvector: given an eigenvalue a of T with corresponding one-dimensional subspace S, a generalized eigenvector of a is a vector x such that (T|S−aI|S)k=0 for some k∈N. Which is to say, subtracting a times the identity operator from T doesn't have to yield the null operator immediately, but it does have to yield the null operator if the resulting operator is applied several times.

In our example above, only vectors of the form (0,x) are eigenvectors, but all vectors are generalized eigenvectors, because T−1I is the operator whose matrix wrt to the standard basis is [0100], and this operator equals the null operator if it is applied twice, i.e. (T−1I)2=0.

If all of the above is done properly, then one gets something that comes very close to the perfect decomposition for any operator on a complex vector space. This is the theory behind the famous Jordan form for matrices: it is not quite a diagonal matrix (which corresponds to the perfect decomposition), but it almost is. It just has a bunch of additional 1s to deal with the generalized eigenvectors.

Chapter 9 then does a similar thing for real vector spaces. As always, it's more cumbersome and the result is weaker, but it's still quite strong.

At last, Chapter 10 introduces determinants, and it's actually kind of boring! All of the interesting LA has already been done, so this chapter feels like a mere afterthought. Again, one doesn't seem to need determinants to do basic LA.

The next big item on my list is Understanding Machine Learning, which is also from Miri's guide. I've so far neglected trying to get familiar with actual AI research, and it's time to remedy that, or at least get a solid grasp of the basics.

Discuss

### No nonsense version of the "racial algorithm bias"

Новости LessWrong.com - 13 июля, 2019 - 18:39
Published on July 13, 2019 3:39 PM UTC

In discussions of algorithm bias, the COMPAS scandal has been too often quoted out of context. This post gives the facts, and the interpretation, as quickly as possible. See this for details.

The fight

The COMPAS system is a statistical decision algorithm trained on past statistical data on American convicts. It takes as inputs features about the convict and outputs a "risk score" that indicates how likely the convict would reoffend if released.

In 2016, ProPublica organization claimed that COMPAS is clearly unfair for blacks in one way. Northpointe replied that it is approximately fair in another way. ProPublica rebukes with many statistical details that I didn't read.

The basic paradox at the heart of the contention is very simple and is not a simple "machines are biased because it learns from history and history is biased". It's just that there are many kinds of fairness, each may sound reasonable, but they are not compatible in realistic circumstances. Northpointe chose one and ProPublica chose another.

The math

The actual COMPAS gives a risk score from 1-10, but there's no need. Consider the toy example where we have a decider (COMPAS, a jury, or a judge) judging whether a group of convicts would reoffend or not. How well the decider is doing can be measured in at least three ways:

• False negative rate = (false negative)/(actual positive)
• False positive rate = (false positive)/(actual negative)
• Calibration = (true positive)/(test positive)

A good decider should have false negative rate close to 0, false positive rate close to 0, and calibration close to 1.

Visually, we can draw a "square" with four blocks:

• false negative rate = the "height" of the false negative block,
• false positive rate = the "height" of the false positive block,
• calibration = (true positive block)/(total area of the yellow blocks)

Now consider black convicts and white convicts. Now we have two squares. Since they have different reoffend rates for some reason, the central vertical line of the two squares are different.

The decider tries to be fair by making sure that the false negative rate and false positive rates to be the same in both squares, but then it will be forced to make the calibration in the Whites lower than the calibration in the Blacks.

Then suppose the decider try to increase the calibration in the Whites, then the decider must somehow decrease the false negative rate of Whites, or the false positive rate of Whites.

In other words, when the base rates are different, it's impossible to have equal fairness measures in:

• false negative rate
• false positive rate
• calibration

In the jargon of fairness measurement, "equal false negative rate and false positive rate" is "parity fairness"; "equal calibration" is just "calibration fairness".

Parity fairness and calibration fairness can be straightforwardly generalized for COMPAS, which uses a 1-10 scoring scale.

The fight, after-math

Northpointe showed that COMPAS is approximately fair in calibration for Whites and Blacks. ProPublica showed that COMPAS is unfair in parity.

The lesson is that their are incompatible fairnesses. To figure out which to apply -- that is a different question.

Discuss

### Reclaiming Eddie Willers

Новости LessWrong.com - 13 июля, 2019 - 18:32
Published on July 13, 2019 3:32 PM UTC

[Content: personal ramble, not trying to be rigorous]

When I read Atlas Shrugged a few years ago, it was one of the more intensely disturbing experiences I’ve had.

I remember that Eddie Willers was the only character I resonated or identified with much. He’s also, as far as I can tell, the only (even slightly positively portrayed) Hufflepuff character in the story. And the last we see is of him alone in the wilderness, as the last train breaks down – mistakenly loyal to the train company, an entity that isn’t capable of loyalty in return, and not agenty or cool enough to join the main protagonists in their escape from the collapse of civilization.

That...really got to me. I won’t make any claims about whether Atlas Shrugged is a particularly well-written book, or whether it even contains this message on purpose, but at that moment in my life, it painted a very vivid, compelling picture of a world in which to be Hufflepuff is to be unsafe, useless, unwanted. Incapable of agency or of doing the right thing when it matters. Eddie is an earnest idealist, trying to do his best by Dagny Taggart and her company, and that trait is his doom.

(I was recently quoted a friend of mine saying “a Hufflepuff among Slytherins will die as surely as among [snakes? Don’t remember exact quote]”. Right now, this feels like an example of that phenomenon.)

I notice a desire to push back against that interpretation. I claim that Eddie is flawed, imperfect, and his last choice ends up being ineffective, but not because of his earnest idealism. He’s being unstrategic, not paying attention to the patterns of his world and what will actually work – but I refuse to say that his caring about the train reaching its destination is a mistake.

Loyalty isn’t necessarily strategic, and blind loyalty can lead into disaster, but I refuse to say that having a drive towards it is inevitably a character flaw.

In the real world, it matters if trains reach their destinations. It’s a bad thing if civilization collapses because all the people who could have stopped it walked away. And it doesn’t make someone a fool, or pitiable, or merely a foil for the true protagonists, if they genuinely and earnestly care.

If I were in Eddie Willers’ shoes, transplanted as I am now into the world of Atlas Shrugged – I don’t think I would be blindly loyal to Dagny Taggart, or to her company. I hope I would actually take a step back, take my feelings of loyalty as object, and reflect on what mattered according to my values and how to protect those things as strategically as possible.

Still, I almost hope my journey would come to the same place – stranded in the wilderness on a broken-down train, because I refused to abandon society’s last hope even as everything crumbled around me.

I refuse to be ashamed of that. And, well, it doesn’t have to end there. The scene might fade to black – but afterward, even if I eventually gave up on that specific train and set it aside as a lost cause, I hope I would pack up my belongings and and start walking. Not away, but towards wherever I could keep trying to help.

In the world of Atlas Shrugged, maybe I would be truly alone in that, and thus doomed to failure. In the real world, our world… I don’t think so. I can be an earnest idealist, deep down, and I’m not the only one.

Discuss

### Let's Read: Superhuman AI for multiplayer poker

Новости LessWrong.com - 13 июля, 2019 - 17:28
Published on July 13, 2019 2:28 PM UTC

On July 11, a new poker AI is published in Science. Called Pluribus, it plays 6-player No-limit Texas Hold'em at superhuman level.

In this post, we read through the paper. The level of exposition is between the paper (too serious) and the popular press (too entertaining).

Basics of Texas Hold'em

If you don't know what it even is, like me, then playing a tutorial would be best. I used Learn Poker on my phone.

Now that you know how to play it, it's time to deal with some of the terminologies.

• No-limit: you can bet as much as you want.
• Limping: betting the minimal amount that you have to bet, in order to keep yourself in the game. This is generally considered bad: if you feel confident, you should raise the bet, and if you feel diffident, you should quit.
• Donk betting: some kind of uncommon play that's usually considered dumb (like a donkey). I didn't figure out what it actually means.
The authors

The authors are Noam Brown and Tuomas Sandholm. Previously, they made the news by writing Libratus, a poker AI that beat human champions in 2-player no-limit Texas Hold'em, in 2017.

Pluribus contains a lot of the code from Libratus and its siblings:

The authors have ownership interest in Strategic Machine, Inc. and Strategy Robot, Inc. which have exclusively licensed prior game-solving code from Prof. Sandholm’s Carnegie Mellon University laboratory, which constitutes the bulk of the code in Pluribus.

Scroll to the bottom for more on the two companies.

Highlights from the paper Is Nash equilibrium even worthwhile?

In multiplayer games, Nash equilibriums are not easy to compute, and might not even matter. Consider the Lemonade Stand Game:

It is summer on Lemonade Island, and you need to make some cash. You decide to set up a lemonade stand on the beach (which goes all around the island), as do two others. There are twelve places to set up around the island like the numbers on a clock. Your price is fixed, and all people go to the nearest lemonade stand. The game is repeated. Every night, everyone moves under cover of darkness (simultaneously). There is no cost to move. After 100 days of summer, the game is over. The utility of the repeated game is the sum of the utilities of the single-shot games.

The Nash equilibrium is when three of you are equidistant from each other, but there's no way to achieve that unilaterally. You might decide that you will just stay in Stand 0 and wait for the others to get to Stand 4 and Stand 8, but they might decide upon a different Nash equilibrium.

The authors decided to go all empirical and not consider the problem of Nash equilibrium:

The shortcomings of Nash equilibria outside of two-player zero-sum games, and the failure of any other game-theoretic solution concept to convincingly overcome them, have raised the question of what the right goal should even be in such games. In the case of six-player poker, we take the viewpoint that our goal should not be a specific game-theoretic solution concept, but rather to create an AI that empirically consistently defeats human opponents, including elite human professionals.

The success of Pluribus shows appears to vindicate them:

... even though the techniques do not have known strong theoretical guarantees on performance outside of the two-player zero-sum setting, they are nevertheless capable of producing superhuman strategies in a wider class of strategic settings.

Description of Pluribus

Pluribus first produces a "blueprint" by offline self-play, then during live gaming, adapt it:

The core of Pluribus’s strategy was computed via self play, in which the AI plays against copies of itself, without any data of human or prior AI play used... Pluribus’s self play produces a strategy for the entire game offline, which we refer to as the blueprint strategy. Then during actual play against opponents, Pluribus improves upon the blueprint strategy by searching for a better strategy in real time for the situations it finds itself in during the game.

Since the first round (like chess opening vs chess midgame) had the smallest amount of variation, Pluribus could afford to train an almost complete blueprint strategy for the first round. For later rounds, some real-time search was needed:

Pluribus only plays according to this blueprint strategy in the first betting round (of four)... After the first round, Pluribus instead conducts real-time search to determine a better, finer-grained strategy for the current situation it is in.

Pluribus uses Monte Carlo counterfactual regret minimization. The details can be found in the link.

The blueprint strategy in Pluribus was computed using a variant of counterfactual regret minimization (CFR)... We use a form of Monte Carlo CFR (MCCFR) that samples actions in the game tree rather than traversing the entire game tree on each iteration.

Pluribus can be sneaky:

... if the player bets in [a winning] situation only when holding the best possible hand, then the opponents would know to always fold in response. To cope with this, Pluribus keeps track of the probability it would have reached the current situation with each possible hand according to its strategy. Regardless of which hand Pluribus is actually holding, it will first calculate how it would act with every possible hand, being careful to balance its strategy across all the hands so as to remain unpredictable to the opponent. Once this balanced strategy across all hands is computed, Pluribus then executes an action for the hand it is actually holding.

This was corroborated by a comment from a human opponent:

"Pluribus is a very hard opponent to play against," said Chris Ferguson, a World Series of Poker champion. "It's really hard to pin him down on any kind of hand."

Scroll down for how Ferguson lost to Pluribus.

Pluribus is cheap, small, and fast

In order to make Pluribus small, the blueprint strategy is "abstracted", that is, it intentionally confuses some game actions (because really, $200 and$201 are not so different).

We set the size of the blueprint strategy abstraction to allow Pluribus to run during live play on a machine with no more than 128 GB of memory while storing a compressed form of the blueprint strategy in memory.

The abstraction paid off. Pluribus was cheap to train, cheap to run, and faster than humans:

The blueprint strategy for Pluribus was computed in 8 days on a 64-core server for a total of 12,400 CPU core hours. It required less than 512 GB of memory. At current cloud computing spot instance rates, this would cost about $144 to produce. When playing, Pluribus runs on two Intel Haswell E5-2695 v3 CPUs and uses less than 128 GB of memory. For comparison... Libratus used 100 CPUs in its 2017 matches against top professionals in two-player poker. On Amazon right now, Intel® Xeon® Processor E5-2695 v3 CPU cost just$500 each, and a 128 GB RAM cost $750. The whole setup can be constructed for under$2000. It would only take a little while to recoup the cost if it goes to online poker.

The amount of time Pluribus takes to conduct search on a single subgame varies between 1 s and 33 s depending on the particular situation. On average, Pluribus plays at a rate of 20 s per hand when playing against copies of itself in six-player poker. This is roughly twice as fast as professional humans tend to play.

Pluribus vs Human professionals. Pluribus wins!

We evaluated Pluribus against elite human professionals in two formats: five human professionals playing with one copy of Pluribus (5H+1AI), and one human professional playing with five copies of Pluribus (1H+5AI). Each human participant has won more than $1 million playing poker professionally. Professional Poker is an endurance game, like marathon: In this experiment, 10,000 hands of poker were played over 12 days. Each day, five volunteers from the pool of [13] professionals were selected to participate based on availability. The participants were not told who else was participating in the experiment. Instead, each participant was assigned an alias that remained constant throughout the experiment. The alias of each player in each game was known, so that players could track the tendencies of each player throughout the experiment. And there was prize money, of course, for the humans. Pluribus played for free -- what a champ.$50,000 was divided among the human participants based on their performance to incentivize them to play their best. Each player was guaranteed a minimum of $0.40 per hand for participating, but this could increase to as much as$1.60 per hand based on performance.

Pluribus had a very high win rate, and is statistically demonstrated to be profitable when playing against 5 elite humans:

After applying AIVAT, Pluribus won an average of 48 mbb/game (with a standard error of 25 mbb/game). This is considered a very high win rate in six-player no-limit Texas hold’em poker, especially against a collection of elite professionals, and implies that Pluribus is stronger than the human opponents. Pluribus was determined to be profitable with a p-value of 0.028.

"mbb/game" means "milli big blinds per game". "big blind" just means "the least amount that one must bet at the beginning of the game", and poker players use it as a unit of measurement of the size of bets. "milli" means 1/1000. So Pluribus would on average win 4.8% of the big blind each game. Very impressive.

AIVAT is statistical technique that is designed specifically to evaluate how good a poker player is. From (Neil Burch et al, 2018):

Evaluating agent performance when outcomes are stochastic and agents use randomized strategies can be challenging when there is limited data available... [AIVAT] was able to reduce the standard deviation of a Texas hold’em poker man-machine match by 85% and consequently requires 44 times fewer games to draw the same statistical conclusion. AIVAT enabled the first statistically significant AI victory against professional poker players in no-limit hold’em.

Pluribus vs Jesus (and Elias)

The human participants in the 1H+5AI experiment were Chris “Jesus” Ferguson and Darren Elias. Each of the two humans separately played 5,000 hands of poker against five copies of Pluribus.

Pluribus did not gang up on the poor human:

Pluribus does not adapt its strategy to its opponents and does not know the identity of its opponents, so the copies of Pluribus could not intentionally collude against the human player.

The humans were paid on average $0.60 per game: To incentivize strong play, we offered each human$2,000 for participation and an additional \$2,000 if he performed better against the AI than the other human player did.

Pluribus won!

For the 10,000 hands played, Pluribus beat the humans by an average of 32 mbb/game (with a standard error of 15 mbb/game). Pluribus was determined to be profitable with a p-value of 0.014.

Ferguson lost less than Elias:

Ferguson’s lower loss rate may be a consequence of variance, skill, and/or the fact that he used a more conservative strategy that was biased toward folding in unfamiliar difficult situations.

Pluribus is an alien, like AlphaZero

And like AlphaZero, it confirms some human strategies, and dismisses some others:

Because Pluribus’s strategy was determined entirely from self-play without any human data, it also provides an outside perspective on what optimal play should look like in multiplayer no-limit Texas hold’em.

Two examples in particular:

Pluribus confirms the conventional human wisdom that limping (calling the “big blind” rather than folding or raising) is suboptimal for any player except the “small blind” player... While Pluribus initially experimented with limping... it gradually discarded this action from its strategy as self play continued. However, Pluribus disagrees with the folk wisdom that “donk betting” (starting a round by betting when one ended the previous betting round with a call) is a mistake; Pluribus does this far more often than professional humans do.

Too dangerous to be released, again

The program is not released for some kind of unspecified risk. (News articles made it specifically about the risk of wrecking the online gambling industry.)

Because poker is played commercially, the risk associated with releasing the code outweighs the benefits. To aid reproducibility, we have included the pseudocode for the major components of our program in the supplementary materials.

Useful quotes from other news report

From Ars Technica:

Pluribus actually confirmed one bit of conventional poker-playing wisdom: it's just not a good idea to "limp" into a hand, that is, calling the big blind rather than folding or raising. The exception, of course, is if you're in the small blind, when mere calling costs you half as much as the other players.

Pluribus placed donk bets far more often than its human opponents... Pluribus makes unusual bet sizes and is better at randomization. "Its major strength is its ability to use mixed strategies... to do this in a perfectly random way and to do so consistently. Most people just can't."

Sandholm cites multi-party negotiation or pricing—such as Amazon, Walmart, and Target trying to come up with the most competitive pricing against each other—as a specific application. Optimal media spending for political campaigns is another example, as well as auction bidding strategies.

There are a bit of details to the two companies of Sandholm:

Sandholm has already licensed much of the poker technology developed in his lab to two startups: Strategic Machine and Strategy Robot. The first startup is interested in gaming and other entertainment applications; Strategy Robot's focus is on defense and intelligence applications.

"Better computer games"... hm, sounds suspiciously nonspecific.

Brown says Facebook has no plans to apply the techniques developed for six-player poker, although they could be used to develop better computer games.

Discuss

### Job description for an independent AI alignment researcher

Новости LessWrong.com - 13 июля, 2019 - 12:47
Published on July 13, 2019 9:47 AM UTC

This is the job description that I've written for myself in order to clarify what I'm supposed to be doing.

I'm posting it here in order to get feedback on my understanding of the job. Also, if you're thinking of becoming an independent researcher, you might find it useful to know what it takes.

Job Title: Independent AI alignment researcher

Location: anywhere (in my case: Kagoshima, Japan)

Reports To: nobody (in a sense: funders, mentors, givers of feedback)

Position Status: not applicable

Responsibilities
• Define AI alignment research projects. Includes finding questions, gauging their significance and devising ways to answer them.
• Execute research projects by reading, thinking and experimenting.
• Write and publish results in the form of blog entries, contributions to discussions and conferences (conference paper, presentation, poster), journal articles, public datasets, software.
• Solicit feedback and use it to improve processes and results.
• Find potential junior (in the sense of (slightly) less experienced in the field) researchers and help them grow.
• Help other researchers with their work.
• Make sure that the money doesn't run out.
• Any other activities required by funders or givers of feedback.
Hiring requirements

Entry level:

• Strong desire to do good for the world by contributing to AI alignment.
• Undergrad degree or equivalent skill level in computer science, maths or machine learning. Includes having researched, written and presented a scientific paper or thesis.
• Ability to define, plan and complete novel projects with little supervision.
• Ability to collaborate remotely.
• Initiative.
• Discipline.
• Ability to speak and write clearly.
• Ability to identify and close gaps in knowledge or skills.
• Ability to write job or funding applications.
• Ability to figure out things that are usually taken care of for employees: taxes, insurance, payments, bookkeeping, budgeting.
• Ability to deal with uncertainty and financial stress.
Resources used

Discuss

### Raw Post: Talking With My Brother

Новости LessWrong.com - 13 июля, 2019 - 05:57
Published on July 13, 2019 2:57 AM UTC

The Circumstances

I am sitting down to write this immediately after one of the most honest conversations I’ve ever had with my brother. The reason I’m posting it to LessWrong is because I think it is a case study in rationality, emotion, personality, political partisanship, and methods of conversation and debate, all topics that are of interest to segments of this community. We spoke for about an hour, first while driving, and then for a long time at the curb.

We started talking about my brother’s interest in getting involved with the local socialist party. He is not the most talkative person, and is a deeply thoughtful and very well-read person. One of his strong interests is in politics and economics, so I decided to ask him about his thoughts on socialism. I am no lassaiz-faire capitalist, but my political preferences are for well-regulated free markets.

Rising Tension

Our conversation became tense quickly. As I tried to ask him critical questions in a neutral, genuine, and thoughtful manner, he would dismiss them using words like “silly,” “stupid,” “artificial binary,” “lack of imagination,” and so on. This didn’t feel good, but I continued, because my hope was that by maintaining my composure and demonstrating repeatedly that I was really listening and responding with my valid questions and concerns, he would see that I really wanted to engage with him and wasn’t trying to shut him down. I used techniques like trying to find our cruxes of disagreement, framing them as respectfully and clearly as I could, but he would swat them down. He grew audibly angrier as the car ride went along.

I could have tried to divert the conversation to some other topic, but I don’t think that’s healthy, and our family dynamic is such that I feel very confident that this would not have led to a happy atmosphere, but to unaddressed simmering resentment that would have lingered beyond our car ride. So I pressed on, all the way until we got to Seattle.

When he accused me of silliness, I offered what I thought was a detailed and thoughtful description of how I thought things might go under his proposed system. When he accused me of disingenuously demanding that every minor detail be worked out in advance to stifle a basic and obviously good shift that needs to happen, I told him that this was my attempt to really put my imagination to work, thinking through the implications of an idea with which I was not entirely familiar. When I simplified my concern in order to deal with his objection that I was overthinking things, he told me that I was painting an oversimplified binary.

Emotional Honesty

It seemed like nothing could please him, and when we got to our destination, I finally told him so. I said in as kindly a way as I could that I love him, respect him, and was only holding this conversation because it’s clearly an important part of his life, and that while it’s OK for him to feel how he feels and think what he thinks, I felt like he was treating me with contempt, and that it seemed like he was trying to shut down questions. I told him that if someone was proposing a massive change in our social system, I would want to understand the details. For me, the evidence that our present system is working tolerably well is all around me, while this proposal is all on paper. It makes sense to me that we would ask for considerable thought and detail before accepting such a wholesale change.

He explained that for him, his anger over the state of American and world politics has been growing over the last few years. To give an example, he explained that his visceral reaction to hearing liberal arguments against socialism is about as automatic and intense as our reaction to personally-directed racial prejudice ought to be. He doesn’t like how intensely angry he gets, but finds it impossible to speak neutrally about the topic. He has lost faith in rational debate as a way to change minds, and hears so much pro-capitalist argumentation that he feels is disingenuous that he finds it hard to believe it could be coming from a place of sincerity. He knows that there’s a big difference between being wrong and being bad, but he feels that the harm inflicted by capitalism is so great that it tends to obscure the difference on an emotional level.

How He Feels

It helped me understand the bind that he finds himself in, even though I disagree with his economic opinions. He experiences a Catch-22, where nobody will change their minds (or even listen) if he speaks neutrally and rationally, but they’ll dismiss him as a crank if he gets heated. It’s not evil to be wrong, but the harm he perceives in the wrongness around him is so great that he feels morally obligated to point it out, in terms that are strong and direct enough to be potentially offensive. And that itself is an emotional dynamic that is so difficult that it makes it extremely hard to find spaces in his relationships with others to lay it out for other people. My perception was that this seems isolating, although he did not confirm or deny that.

He then offered that if we were to discuss this topic again, it would actually help him keep the tension down if I felt like I could use the same kinds of rude speech to fire right back at him.

How I Feel

I was able to explain that for me, adopting a neutral and rational approach on a topic like this is both a moral duty and an emotional defense mechanism. With a topic this big and important, I feel it’s important to be able to look at it from all sides over a long period of time, and to bring as rigorous a scientific approach as we are able to as a society.

This is one of the topics that has really failed to generate a Kuhnian paradigm revolution with time; there might be a mainstream consensus of capitalist economists, but there are still plenty of people and countries and economists who believe in varieties of socialism, and that’s not just because the old guard hasn’t died yet. Since both sides have a great deal of scholarship behind them, and I’m not an expert, it makes the most sense to choose the argument that makes the most sense, but also leave great room for amicable differences. By contrast, he feels that you’ve got to start by understanding that people simply argue whatever side is in their interests. The first thing to do is pick the side of the victims of injustice, then determine which economic system is primarily looking out for them, and then adhere to that side.

I also told him that when I speak even slightly rudely to people, I immediately become intensely anxious that I’ve upset them, and shut down both socially and intellectually. Furthermore, my attempt at neutral rationality is not a strain or some “elevated tone” for me, but is rather my default state where I feel most natural and relaxed and happy.

Hope for the Future

After talking about that for a while, we were able to see that knowing these things about each other might help us have more open and agreeable conversations with each other in the future. He might feel less of a need to clam up about politics, since he knows that if he comes across very strongly with me, I’ll understand where it’s coming from. I’ll understand that if he gets very heated, it’s not personally directed at me, but is rather an expression of his frustration with the system. And we will hopefully be able to weave in a discussion about how the dynamics of the conversation make us feel, as well as discussing the issues themselves.

Moral and Political Dynamics

This experience helped me shift away from either a “politics is the mindkiller” perspective or a hope that “political conflicts between people with good relationships can be resolved through patient, rational engagement.” Instead, I had to acknowledge that, just as there is no voting system that can possibly achieve all our aims, there is no approach to morality, and therefore economics, that can achieve all our moral aims. Despite that, people will feel intensely passionate about their fundamentally intuitive moral framework. Adopting a neutral, rational approach or a passionate, intense approach to debate can both seem frustrating and disingenuous. Both have their uses.

Conclusion

If the goal is to understand each other, we’ll need to have greater tolerance for our different valences around how we communicate. On some level, even the strongest attempts at holding dialog can easily come across as intensely threatening - not because they’re threatening to demolish your poorly-thought-out ideas, but because they seem to be using neutrality to smother the moral import of the issue.

In order to overcome that dynamic, the only hope is to be able to honestly express how those tense conversations make us feel. We have to be articulate about why we prefer our particular way of speaking, and extend appreciation and sympathy to the other person. If we cannot find common ground in our intellectual beliefs, we can find it in telling the other person that we love them and are doing our best to connect with them, and creating the space for trying to understand not why we disagree but why we’re hurting each other and how to stop.

Discuss

### [Debate] Keeping Beliefs Cruxy and Frames Explicit

Новости LessWrong.com - 13 июля, 2019 - 04:20
Published on July 13, 2019 1:20 AM UTC

Benito and Raemon, from the LessWrong Team just had a discussion about a phrase Ray started saying recently "Keep your Beliefs Cruxy and your Frames Explicit," which Ben felt he probably disagreed with.

After chatting for an hour, Ben started writing his thoughts into a shortform post/comment, and Ray proposed moving it to a dedicated debate post. See below for the discussion.

Discuss

### What are we predicting for Neuralink event?

Новости LessWrong.com - 12 июля, 2019 - 22:33
Published on July 12, 2019 7:33 PM UTC

Interesting exercise in AI-adjacent forecasting area (brain-computer interfaces). Curious if people want to specify some possible reveals+probabilities. https://twitter.com/neuralink/status/1149133717048188929

(if in the somewhat likely scenario you're relying on inside info please mention it)

Discuss

### Largest open collection quotes about AI

Новости LessWrong.com - 12 июля, 2019 - 20:25
Published on July 12, 2019 5:18 PM UTC

I apologize for my bad English, this is not my native language. And probably I will make some mistakes when posting.

For over 2 years I have been reading materials on the topic of AI Safety. I don't have the appropriate education, cognitive abilities, knowledge. I do not even have time to learn the language. So I didn't hope to do something useful myself.
But once I tried to systematize quotations from one show in order to understand when the experts represented there are waiting for AGI and how likely they consider the extinction of humanity.
I thought it would be interesting to do so with the rest of the experts.
In addition, I have already seen and studied with interest such collections of quotes. It seemed to me that the best thing I could do was try to do something similar.

Therefore, I began to collect quotes from people who can be attributed to the experts. It turned out to be harder than I thought.
I have compiled a table with quotes from more than 800 experts. I tried not to distort the opinion of forecasters and simply copied from sources, sometimes deleting or slightly editing. My edits can be recognized by square brackets :)

1) The first column of the table is the name of the expert.
2) The second column is the year of the forecast. The table is built in chronological order.
3) The third column is the predicted time for AGI. Unfortunately, most people did not speak directly about time and probability. Because of this, many quotes came out rather vague. For example, “Machines are very far from being intelligent” or “And we can reach it in a close time”.
4) The fourth column is an opinion about Takeoff Speed. About how much progress will be accelerated after create of AGI.
5) The fifth column is the expert's opinion about the future of mankind with AGI. Choosing a quote here was the hardest. Most of all I was interested in the risk of extinction or serious shocks due to AI, and I tried to provide quotes that most fully reveal this particular topic.
6) The sixth column indicates the source of the quote.

That is, to the right of the forecaster's name, you can find out the date of the given quotes, his opinion about the time of the creation of AI, about the intellectual explosion and about the future of humanity, as well as get acquainted with the source.

Of course, cases where the expert spoke on the topic of time, the speed of self-improvement and the influence of AI in the framework of one material are quite rare. Therefore many cells are left empty.
I had to give several quotes per person, sometimes they were separated for years and even decades.
Since all the quotes are given in chronological order, the opinions of some people are “scattered” in the table.
For example, Gwern spoke about the future of mankind in 2010, about the growth of AI in 2014 and about the forecasts for the emergence of AI in 2018.
However, you can simply use search.

In addition, sometimes one person has already made a certain forecast but later changed or expanded his opinion. I tried to take into account such quotes.

I also reviewed anonymous expert interviews and indicated them. If the general list of respondents was known, I cited them as well.

It was difficult to decide who should be considered an expert and what quotes should be included in the work.
I had to make controversial decisions. The table includes a lot of people who are entrepreneurs but may have insights on advanced research. There are several futurists and philosophers in the table. There are writers like Clark and Vinge, whose opinion seems important to me.

I have a version of this work without chronological separation, where the quotes are more grouped by name. Perhaps someone will be more convenient.

It is difficult to draw conclusions from the work. The absolute majority of experts did not talk about exact dates and did not indicate the probability of their predictions.
I can only say that most forecasters do not expect AI in the near future, do not expect IE and seem optimistic.
In addition, it seemed to me that in the twentieth century the leading experts were on average more pessimistic: Turing, Wiener, I. J. Good, Fredkin, Shannon, Moravec, etc.
Young researchers are on average more optimistic than older ones - even in the field of AI Safety, where on average there are naturally more concerned people.
I think that to confirm almost any views you can find the opinion of a respected expert.

I really hope that for someone my work will be useful and interesting.