Вы здесь

Новости LessWrong.com

Подписка на Лента Новости LessWrong.com Новости LessWrong.com
A community blog devoted to refining the art of rationality
Обновлено: 52 минуты 9 секунд назад

The Amish, and Strategic Norms around Technology

16 часов 46 минут назад
Published on March 24, 2019 10:16 PM UTC

I was reading Legal Systems Very Different From Ours by David Friedman. The chapter on the Amish made a couple interesting claims, which changed my conception of that culture (although I'm not very confident that the Amish would endorse these claims as fair descriptions).

Strategic Norms Around Technology

The Amish relationship to technology is not "stick to technology from the 1800s", but rather "carefully think about how technology will affect your culture, and only include technology that does what you want."

So, electric heaters are fine. Central heating in a building is not. This is because if there's a space-heater in the living room, this encourages the family to congregate together. Whereas if everyone has heating in their room, they're more likely to spend time apart from each other.

Some communities allow tractors, but only if they don't have rubber tires. This makes them good for tilling fields but bad for driving around.

Cars and telephones are particularly important not to allow, because easy transportation and communication creates a slippery slope to full-connection to the outside world. And a lot of the Amish lifestyle depends on cutting themselves off from the various pressures and incentives present in the rest of the world.

Some Amish communities allow people to borrow telephones or cars from non-Amish neighbors. I might have considered this hypocritical. But in the context of "strategic norms of technology", it need not be. The important bit is to add friction to transportation and communication.

Competitive Dictatorship

Officially, most Amish congregations operate via something-like-consensus (I'm not sure I understood this). But Friedman's claim is that in practice, most people tend to go with what the local bishop says. This makes a bishop something like a dictator.

But, there are lots of Amish communities, and if you don't like the direction a bishop is pushing people in, or how they are resolving disputes, you can leave. There is a spectrum of communities ranging in how strict they are about about various rules, and they make decisions mostly independently.

So there is not only strategic norms around technology, but a fairly interesting, semi-systematic exploration of those norms.

Other Applications

I wouldn't want to be Amish-in-particular, but the setup here is very interesting to me.

I know some people who went to MAPLE, a monastery program. While there, there were limits on technology that meant, after 9pm, you basically had two choices: read, or go to bed. The choices were strongly reinforced by the social and physical environment. And this made it much easier to make choices they endorsed.

Contrast this with my current house, where a) you face basically infinite choices about to spend your time, and b) in practice, the nightly choices often end up being something like "stay up till 1am playing minecraft with housemates" or "stay up till 2am playing minecraft with housemates."

I'm interested in the question "okay, so... my goals are not the Amish goals. But, what are my goals exactly, and is there enough consensus around particular goals to make valid choices around norms and technology other than 'anything goes?'"

There are issues you face that make this hard, though:

Competition with the Outside World – The Amish system works because it cuts itself off from the outside world, and its most important technological choices directly cause that. Your business can't get outcompeted by someone else who opens up their shop on Sundays because there is nobody who opens their shop on Sundays.

You also might have goals that directly involve the outside world.

(The Amish also have good relationships with the government such that they can get away with implementing their own legal systems and get exceptions for things like school-laws. If you want to do something on their scale, you both would need to not attract the ire of the government, and be good enough at rolling your own legal system to not screw things up and drive people away)

Lack of Mid-Scale-Coordination – I've tried to implement 10pm bedtimes. It fails, horribly, because I frequently attend events that last till midnight or later. Everyone could shift their entire sleep schedule forward, maybe. But also...

People Are Different – Some of people's needs are cultural. But some are biological, and some needs are maybe due to environmental factors that happened over decades and can't be changed on a dime.

Some people do better with rules and structure. Some people flourish more with flexibility. Some people need rules and structure but different rules and structure than other people.

This all makes it fairly hard to coordinate on norms.

Contenders for Change

Given the above, I think it makes most sense to:

  • Look for opportunities explore norms and technology-use at the level of individuals, households, and small organizations (these seem like natural clusters with small numbers of stakeholders, where you can either get consensus or have a dictator).
  • While doing so, choose norms that are locally stable, that don't require additional cooperation outside yourself, your household or your org.

For example, I could imagine an entire household trying out a rule, like "the household internet turns off at 10pm", or "all the lights turn reddish at night so it's easier to get to sleep"


Did the recent blackmail discussion change your beliefs?

24 марта, 2019 - 19:06
Published on March 24, 2019 4:06 PM UTC

Various rationalist blogs and Less Wrong have recently posted on and discussed blackmail, and specifically legality and acceptability of such. I found the discussion unsatisfying, and I'm trying to understand why that is, and whether I'm alone in that.

As it was happening, it didn't feel like a particularly political topic - nobody seemed personally invested in the outcome. But it did seem like everyone (including myself, sometimes) was presenting examples or (over)generalizing to support their beliefs, and very few were seeking counterexamples or cruxes or lines of demarcation between different intuitions.

So - was this politics in disguise? Was some other bias interfering with the discussion? Was it useful and I just missed it? Did any sort of consensus emerge?


The Politics of Age (the Young vs. the Old)

24 марта, 2019 - 09:40
Published on March 24, 2019 6:40 AM UTC

Few days ago I've read an article in the local newspaper about Switzerland considering to lower the voting age to 16.

The reason I found it interesting was that it was not one of the old tired political discussions supported by the same old tired arguments that you typically encounter. In fact, it's a question that I have never thought of before.

Apparently, the discussion was triggered by the recent school strike for climate that went quite big in Switzerland. I've attended the demonstration in Zurich and it was not only big, it was really a kids' event. You could spot a grown-up here and there but they were pretty rare. (Btw, I think this movement is worth watching. Here, for the first time, I see a coordination on truly global level. It spans beyong western countries, with events being hosted in Asia, Pacific Islands, South America or Africa.)

Anyway, the main argument for lowering the voting age is to counter-balance the greying of the electorate.

Once again, this stems from what the climate stikers say: "The politicians who decide on these issues will be dead by the time the shit hits the fan. It will be us who'll have to deal with it. We should have a say in the matter."

But the question is broader: As the demographics change, with the birth rates dropping at crazy speed (China's population will start shrinking not that far in the future; Sub-saharan fartility rates had plummeted from 6.8 in 1970's to 4.85 in 2015), the age pyramid is going to look less like a pyramid and more like a column or even a funnel. In such a case the old will hold a much larger amount of political power than they do today.

While that may seem like a minor thing (everyone is young at some point and old later on) just consider how it would affect the politics of, say, pensions or health-care.

Or, for that matter, I hear that Brexit wouldn't happen is 16- and 17-year olds were allowed to vote.

More questions:

With old people being generally more conservative are we going to see slowing or even reversal of the seemingly instoppable move to the political left that was going on for decades?

With high percentage of young males being often blamed for social unrest and wars, is the changing shape of the age pyramid going to result in even more political stability? And how is giving teenagers a vote going to affect that?

I have no answers but the topic is definitely worth thinking about.

(Btw, the voting age was lowered to 16 in canton Glarus in 2007, so there's more than a decade of data to analyse the impact of the measure.)

March 24th, 2019

by martin_sustrik


Why the AI Alignment Problem is Unsolvable

24 марта, 2019 - 07:10
Published on March 24, 2019 4:10 AM UTC

The following is a chapter from the story I've been writing which contains a proof I came up with that the value alignment problem is unsolvable. I know it sounds crazy, but as far as I can tell the proof is completely correct. There are further supporting technical details which I can explain if anyone asks, but I didn't want to overload you guys with too much information at once, since a lot of those additional supporting details would require articles of their own to explain.

I am not the first person to make a correct proof that the Value Alignment problem is unsolvable. The credit for that goes to my friend Exceph, who came up with a longer and more technical proof which involves content from the Sequence on Instrumental Rationality we've been working on. His proof has not been published yet.

I haven't had time yet to extract my own less technical proof from the narrative dialogue of my story, but I thought it was really important that I share it here as soon as possible, since the more time is wasted on AI research, the less time we have to come up with strategies and solutions that could more effectively prevent x-risk long term.

Also, HEAVY SPOILERS for the story I've been writing, Earthlings: People of the Dawn. This chapter is literally the last chapter of part 5, after which the remaining parts are basically extended epilogues. You have been warned.



There were guards standing outside the entrance to the Rationality Institute. They saluted Bertie as he approached. Bertie nodded to them as he walked past. He reached the front doors and turned the handle, then pulled the door open.

He stepped inside. There was no one at the front desk. All the lights were on, but he didn’t hear anyone in the rooms he passed as he walked down the hallway, approaching the door at the end.

He finally stood before it. It was the door to Thato’s office.

Bertie knocked.

“Come in,” he heard Thato say from the other side.

Bertie turned the knob with a sweaty hand and pushed inwards. He stepped inside, hoping that whatever Thato wanted to talk to him about, that it wasn’t an imminent existential threat.

“Hello Bertie,” said Thato, somberly. He looked sweaty and tired, with bags under his puffy red eyes. Had he been crying?

“Hi Thato,” said Bertie, gently shutting the door behind him. He pulled up a chair across from Thato’s desk. “What did you want to talk to me about?”

“We finished analyzing the research notes on the chip you gave us two years ago,” said Thato, dully.

“And?” asked Bertie. “What did you find?”

“It was complicated, it took us a long time to understand it,” said Thato. “But there was a proof in there that the value alignment problem is unsolvable.”

There was a pause, as Bertie’s brain tried not to process what it had just heard. Then…

“WHAT!?” Berite shouted.

“We should have realized it earlier,” said Thato. Then in an accusatory tone, “In fact, I think you should have realized it earlier.”

“What!?” demanded Bertie. “How? Explain!”

“The research notes contained a reference to a children's story you wrote: A Tale of Four Moralities,Thato continued, his voice rising.It explained what you clearly already knew when you wrote it, that there are actually FOUR types of morality, each of which has a different game-theoretic function in human society: Eye for an Eye, the Golden Rule, Maximize Flourishing and Minimize suffering.”

“Yes,” said Bertie. “And how does one go from that to ‘the Value Alignment problem is unsolvable’?”

“Do you not see it!?” Thato demanded.

Bertie shook his head.

Thato stared at Bertie, dumbfounded. Then he spoke slowly, as if to an idiot.

“Game theory describes how agents with competing goals or values interact with each other. If morality is game-theoretic by nature, that means it is inherently designed for conflict resolution and either maintaining or achieving the universal conditions which help facilitate conflict resolution for all agents. In other words, the whole purpose of morality is to make it so that agents with competing goals or values can coexist peacefully! It is somewhat more complicated than that, but that is the gist.”

“I see,” said Bertie, his brows furrowed in thought. “Which means that human values, or at least the individual non-morality-based values don’t converge, which means that you can’t design an artificial superintelligence that contains a term for all human values, just the moral values.”

Then Bertie had a sinking, horrified feeling accompanied by a frightening intuition. He didn’t want to believe it.

“Not quite,” said Thato cuttingly. “Have you still not realized? Do you need me to spell it out?”

“Hold on a moment,” said Bertie, trying to calm his racing anxiety.

What is true is already so, Bertie thought.

Owning up to it doesn’t make it worse.

Not being open about it doesn’t make it go away.

And because it’s true, it is what is there to be interacted with.

People can stand what is true, for they are already enduring it.

Bertie took a deep breath as he continued to recite in his mind…

If something is true, then I want to believe it is true.

If something is not true, then I want not to believe it is true.

Let me not become attached to beliefs I may not want.

Bertie exhaled, still overwhelmingly anxious. But he knew that putting off the revelations any longer would make it even harder to have them. He knew the thought he could not think would control him more than the thought he could. And so he turned his mind in the direction it was afraid to look.

And the epiphanies came pouring out. It was a stream of consciousness, no--a waterfall of consciousness that wouldn’t stop. Bertie went from one logical step to the next, a nearly perfect dance of rigorously trained self-honesty and common sense--imperfect only in that he had waited so long to start it, to notice.

“So you can’t program an intelligence to be compatible with all human values, only human moral values,” Bertie said in a rush. “Except even if you programmed it to only be compatible with human moral values, there are four types of morality, so you’d have four separate and competing utility functions to program into it. And even if somehow you could program an intelligence to optimize for those four competing utility functions at the same time, that would just cause it to optimize for conflict resolution, and then it would just tile the universe with tiny artificial conflicts between artificial agents for it to resolve as quickly and efficiently as possible without letting those agents do anything themselves.”

“Right in one,” said Thato with a grimace. “And as I am sure you already know, turning a human into a superintelligence would not work either. Human values are not sufficiently stable. If you instruct a superintelligent human to protect other humans from death or grievous injury without infringing on their self-determination, that human would by definition have to stay out of human affairs under most circumstances, only intervening to prevent atrocities like murder, torture or rape, or to deal with the occasional existential threat. It would eventually go mad with boredom and loneliness, and it would snap.

“So, to summarize,” Bertie began, slowly. “The very concept of an omnibenevolent god is a contradiction in terms. It doesn’t correspond to anything that could exist in any self-consistent universe. It is logically impossible.”

“Hindsight is twenty-twenty, is it not?” asked Thato rhetorically.


“So what now?” asked Bertie.

“What now?” repeated Thato. “Why, now I am going to spend all of my money on frivolous things, consume copious amounts of alcohol, say anything I like to anyone without regard for their feelings or even safety or common sense, and wait for the end. Eventually, likely soon, some twit is going to build a God, or blow up the world in any number of other ways. That is all. It is over. We lost.”

Bertie stared at Thato. Then in a quiet, dangerous voice he asked, “Is that all? Is that why you sent me a message saying that you urgently wanted to meet with me in private?”

“Surely you see the benefit of doing so?” asked Thato. “Now you no longer will waste any more time on this fruitless endeavor. You too may relax, drink, be merry and wait for the end.”

At this point Bertie was seething. In a deceptively mild tone he asked, “Thato?”

“Yes?” asked Thato.

“May I have permission to slap you?”

“Go ahead,” said Thato. “It does not matter anymore. Nothing does.”

Bertie leaned over the desk and slapped Thato across the face, hard.

Thato seized Bertie’s wrist and twisted it painfully.

“That bloody hurt, you git!”

“I thought you said nothing matters!?” Bertie demanded. “Yet it clearly matters to you whether you’re slapped.”

Thato released Bertie’s wrist and looked away. Bertie massaged his wrist, trying to make the lingering sting go away.

"Are you done being an idiot?" he asked.

"Define 'idiot'," said Thato scathingly, still not looking at him.

"You know perfectly well what I mean," said Bertie.

Thato ignored him.


Bertie clenched his fists.

“In the letter Yuuto gave me before he died, he told me that the knowledge contained in that chip could spell Humanity’s victory or its defeat,” he said angrily, eyes blazing with determination. “Do you get it? Yuuto thought his research could either destroy or save humankind. He wouldn’t have given it to me if he didn’t think it could help. So I suggest you and your staff get back to analyzing it. We can figure this out, and we will.”

Bertie turned around and stormed out of the office.

He did not look back.


A Tale of Four Moralities

24 марта, 2019 - 06:46
Published on March 24, 2019 3:46 AM UTC

Author's note: This is a children's story I wrote a while back, which teaches a very important life lesson that none of us got to learn as kids. That lesson is extremely important, so all the adults here should pay attention too. I'll explain more of the technical details of the underlying theory behind it later.


Ivan was very angry.
His teddy was stolen.

Ivan decided.
He would catch the thief and steal from them.

"This will pay them back," said Ivan. "Serves them right."

Goldie was very happy.
It was her birthday.
Her papa gave her a teddy.

Goldie decided.
She would give a gift to her papa in return.

"It was nice of him to give me a teddy," said Goldie.
"This is the least I can do."

The next day, her teddy was gone.

Minnie was very sad. Someone was stealing teddies from her friends.
She looked at her teddy.
Would she be next?

Minnie decided. She would find the stolen teddies.
And she would return them.

"It's the right thing to do," said Minnie.
"This way, no one will be missing their teddies. Not anymore."

The next day, her teddy was gone.

Maxie felt guilty, but hopeful.
Earlier, his mama told him something sad.

"The other neighborhood is poor.
Kids there don't have teddies."

So Maxie decided.
He would steal teddies from his friends. He would give them to the other neighborhood.

"It's the best thing I can do," said Maxie. "My friends can afford new teddies. But the poor kids can't."

So Maxie stole teddies from his friends,
and gave them to the other neighborhood.

This made the kids there happy.
But his friends were sad, because now THEY had no teddies.

The next day,
the sad kids went with their parents to the teddy store,
to buy them new teddies.
But the store was all sold out of teddies.

"It's been hard to sell teddies in this town," said the store clerk. "Many poor people can't afford them. And many rich people already have teddies."

"Why not give teddies to the poor?
For free?" asked Maxie.

"We tried that before," said the clerk.
"It didn't work.
A long line of people came for teddies.
Many poor people can't afford cars.
When they got here, they were last in line. Then they got to the front of the line.
But by then, we were out of teddies."

"Then why give teddies to the rich?" asked Minnie.
"Can't you tell them no?"

"Other rich people paid us to give teddies for free.
They can't do that all the time.

"We have to sell to the rich, too.
Otherwise, we can't afford to make teddies.

"At all."

"Why not?" asked Goldie.

"We have to pay for the stuff to make the teddy," said the clerk.

"Why can't you just get that stuff for free?" asked Maxie.
"Then you could give teddies, without being paid."

"Maxie," said Maxie's mama. "There aren't enough teddies for everyone.
There isn't enough stuff to make that many."

Maxie began to cry.
"I wanted to make more people happier," he said.
"I thought by giving teddies to poor kids, I could make more of the town happier. There are more kids in the poor neighborhood.
And they had no teddies."

"YOU stole our teddies!" Ivan accused. "You should be punished.
Someone should steal a teddy from you."

"I'm sorry!" said Maxie.
"I don't have any teddies.
I gave them to the kids in the other neighborhood."

"Maybe if you asked nicely, they would return our teddies?" asked Goldie.

"No," said Minnie.
"They would feel the same way we did, when the teddies were stolen from us.
They don't know the teddies were stolen.
If we tell them, they won't know we're telling the truth."

No one was sure what to do.

Finally, Maxie said,
"We need to find a way to make more stuff.
That way, there will be enough to make teddies for everyone."

"And if we can't do that?" asked Minnie.

"I don't know," said Maxie.
"But we have to try!"

"Why should we help everyone?
The poor kids have never helped us," said Goldie.

"What else can we do?" asked Minnie. "We can't steal the teddies back."

"The poor kids didn't do anything wrong!" said Ivan. "We shouldn't punish them!"

"Maybe if we find a way to make more stuff," said Maxie.
"the poor kids will have enough to give you something, in return."

"Okay," said Goldie. "I'll help."

The kids talked.

The parents looked at each other.

"Do you think they can do it?" asked Goldie's papa.

Ivan's mama laughed.
She thought it was a joke.

Minnie's papa sighed sadly.

And Maxie's mama turned to the kids and said:

"If you're kind and just,
understanding and giving.
If you listen to each other, and to others.
If you work hard and do your best.
If you learn, grow and become stronger.
If you are brave, and never give up.
Then, maybe, you will find a way."

They would find a way to make more stuff. Someday.

And so they began their quest.


800 scientist call out against statistical significance

23 марта, 2019 - 15:57

Willing to share some words that changed your beliefs/behavior?

23 марта, 2019 - 05:08
Published on March 23, 2019 2:08 AM UTC

I'm collecting data on powerfully persuasive speech acts; it's part of a dangling thread of curiosity after GPT-2 (a new and fairly powerful text generation algorithm). I'm skeptical of the danger of mind-warping sentences as sometimes presented in fiction, or AI scenarios, and trying to get a sense of what the territory is like.

I've made a form to collect personal examples of things-someone-said that caused you to seriously change some belief or behavior. An easy example would be if someone declared that they love you, and this caused you to suddenly devote a lot more (or a lot less!) time and attention to them as a person.

If you have five minutes, my goal for this form is 1000+ responses and your own response(s) will help with that. All replies are anonymous, and there's a place for you to restrict how the information is used/state confidentiality desires. You can also fill it out more than once if you want.

https://goo.gl/forms/39x3vJqNomAome382 is the link to the form, if you want to share with anyone else; I'm happy to have this spread around wherever.


New Entry at the Stanford Encyclopedia of Philosophy on the Pragmatic Theory of Truth

22 марта, 2019 - 21:33
Published on March 22, 2019 5:39 PM UTC


Can a Bayesian agent be infinitely confused?

22 марта, 2019 - 21:02
Published on March 22, 2019 6:02 PM UTC

Eliezer and others talked about how a Bayesian with a 100% prior cannot change their confidence level, whatever evidence they encounter. that's because it's like having infinite certainty. I am not sure if they meant it literary or not (is it really mathematically equal to infinity?), but assumed they do.

I asked myself, well, what if they get evidence that was somehow assigned 100%, wouldn't that be enough to get them to change their mind? In other words -

If P(H) = 100%

And P(E|H) = 0%

than what's P(H|E) equals to?

I thought, well, if both are infinities, what happens when you subtract infinities? the internet answered that it's indeterminate*, meaning (from what i understand), that it can be anything, and you have absolutely no way to know what exactly.

So i concluded that if i had understated everything correct, than such a situation would leave the Bayesian infinitely confused. in a state that he has no idea where he is from 0% to a 100%, and no amount of evidence in any direction can ground him anywhere.

Am i right? or have i missed something entirely?

*I also found out about Riemann's rearrangement theorem which, in a way, let's you arrange some infinite series in a way that equals whatever you want. Dem, that's cool!


The Game Theory of Blackmail

22 марта, 2019 - 20:44
Published on March 22, 2019 5:44 PM UTC

This blog post is composed as following:

  1. Review of Prisoners Dilemma
  2. Explanation of Game of Chicken by comparing it to Prisoners Dilemma
  3. Blackmail is a Game of Chicken
  4. Why we should care about blackmail/Game of Chicken
  5. What to do? Iterated Game of Chicken?

You are encouraged to skip ahead to the part that interests you

1. Review of Prisoners Dilemma

Prisoners Dilemma is a class of two player games which can represent for example mutual beneficial cooperation, or the tragedy of the commons. I don't think it is controversial to say that this class of games are important in almost any multi-agent scenario.

In a Prisoners Dilemma , each player gets to choose between two actions, usually called "cooperate" and "defect". Further more the payoffs haved to fulfill the following:

  • Holding my action constant it is better for me if you cooperate.
  • Holding your action constant, it is better for me if I defect.
  • Cooperate-cooperate is Pareto optimal (even when including mixed strategies).

Example of a payout matrix for Prisoners Dilemma:

.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style: normal; font-weight: normal; font-size: 100%; font-size-adjust: none; letter-spacing: normal; word-wrap: normal; word-spacing: normal; white-space: nowrap; float: none; direction: ltr; max-width: none; max-height: none; min-width: 0; min-height: 0; border: 0; margin: 0; padding: 1px 0} .MJXc-display {display: block; text-align: center; margin: 1em 0; padding: 0} .mjx-chtml[tabindex]:focus, body :focus .mjx-chtml[tabindex] {display: inline-table} .mjx-full-width {text-align: center; display: table-cell!important; width: 10000em} .mjx-math {display: inline-block; border-collapse: separate; border-spacing: 0} .mjx-math * {display: inline-block; -webkit-box-sizing: content-box!important; -moz-box-sizing: content-box!important; box-sizing: content-box!important; text-align: left} .mjx-numerator {display: block; text-align: center} .mjx-denominator {display: block; text-align: center} .MJXc-stacked {height: 0; position: relative} .MJXc-stacked > * {position: absolute} .MJXc-bevelled > * {display: inline-block} .mjx-stack {display: inline-block} .mjx-op {display: block} .mjx-under {display: table-cell} .mjx-over {display: block} .mjx-over > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-under > * {padding-left: 0px!important; padding-right: 0px!important} .mjx-stack > .mjx-sup {display: block} .mjx-stack > .mjx-sub {display: block} .mjx-prestack > .mjx-presup {display: block} .mjx-prestack > .mjx-presub {display: block} .mjx-delim-h > .mjx-char {display: inline-block} .mjx-surd {vertical-align: top} .mjx-mphantom * {visibility: hidden} .mjx-merror {background-color: #FFFF88; color: #CC0000; border: 1px solid #CC0000; padding: 2px 3px; font-style: normal; font-size: 90%} .mjx-annotation-xml {line-height: normal} .mjx-menclose > svg {fill: none; stroke: currentColor} .mjx-mtr {display: table-row} .mjx-mlabeledtr {display: table-row} .mjx-mtd {display: table-cell; text-align: center} .mjx-label {display: table-row} .mjx-box {display: inline-block} .mjx-block {display: block} .mjx-span {display: inline} .mjx-char {display: block; white-space: pre} .mjx-itable {display: inline-table; width: auto} .mjx-row {display: table-row} .mjx-cell {display: table-cell} .mjx-table {display: table; width: 100%} .mjx-line {display: block; height: 0} .mjx-strut {width: 0; padding-top: 1em} .mjx-vsize {width: 0} .MJXc-space1 {margin-left: .167em} .MJXc-space2 {margin-left: .222em} .MJXc-space3 {margin-left: .278em} .mjx-test.mjx-test-display {display: table!important} .mjx-test.mjx-test-inline {display: inline!important; margin-right: -1px} .mjx-test.mjx-test-default {display: block!important; clear: both} .mjx-ex-box {display: inline-block!important; position: absolute; overflow: hidden; min-height: 0; max-height: none; padding: 0; border: 0; margin: 0; width: 1px; height: 60ex} .mjx-test-inline .mjx-left-box {display: inline-block; width: 0; float: left} .mjx-test-inline .mjx-right-box {display: inline-block; width: 0; float: right} .mjx-test-display .mjx-right-box {display: table-cell!important; width: 10000em!important; min-width: 0; max-width: none; padding: 0; border: 0; margin: 0} .MJXc-TeX-unknown-R {font-family: monospace; font-style: normal; font-weight: normal} .MJXc-TeX-unknown-I {font-family: monospace; font-style: italic; font-weight: normal} .MJXc-TeX-unknown-B {font-family: monospace; font-style: normal; font-weight: bold} .MJXc-TeX-unknown-BI {font-family: monospace; font-style: italic; font-weight: bold} .MJXc-TeX-ams-R {font-family: MJXc-TeX-ams-R,MJXc-TeX-ams-Rw} .MJXc-TeX-cal-B {font-family: MJXc-TeX-cal-B,MJXc-TeX-cal-Bx,MJXc-TeX-cal-Bw} .MJXc-TeX-frak-R {font-family: MJXc-TeX-frak-R,MJXc-TeX-frak-Rw} .MJXc-TeX-frak-B {font-family: MJXc-TeX-frak-B,MJXc-TeX-frak-Bx,MJXc-TeX-frak-Bw} .MJXc-TeX-math-BI {font-family: MJXc-TeX-math-BI,MJXc-TeX-math-BIx,MJXc-TeX-math-BIw} .MJXc-TeX-sans-R {font-family: MJXc-TeX-sans-R,MJXc-TeX-sans-Rw} .MJXc-TeX-sans-B {font-family: MJXc-TeX-sans-B,MJXc-TeX-sans-Bx,MJXc-TeX-sans-Bw} .MJXc-TeX-sans-I {font-family: MJXc-TeX-sans-I,MJXc-TeX-sans-Ix,MJXc-TeX-sans-Iw} .MJXc-TeX-script-R {font-family: MJXc-TeX-script-R,MJXc-TeX-script-Rw} .MJXc-TeX-type-R {font-family: MJXc-TeX-type-R,MJXc-TeX-type-Rw} .MJXc-TeX-cal-R {font-family: MJXc-TeX-cal-R,MJXc-TeX-cal-Rw} .MJXc-TeX-main-B {font-family: MJXc-TeX-main-B,MJXc-TeX-main-Bx,MJXc-TeX-main-Bw} .MJXc-TeX-main-I {font-family: MJXc-TeX-main-I,MJXc-TeX-main-Ix,MJXc-TeX-main-Iw} .MJXc-TeX-main-R {font-family: MJXc-TeX-main-R,MJXc-TeX-main-Rw} .MJXc-TeX-math-I {font-family: MJXc-TeX-math-I,MJXc-TeX-math-Ix,MJXc-TeX-math-Iw} .MJXc-TeX-size1-R {font-family: MJXc-TeX-size1-R,MJXc-TeX-size1-Rw} .MJXc-TeX-size2-R {font-family: MJXc-TeX-size2-R,MJXc-TeX-size2-Rw} .MJXc-TeX-size3-R {font-family: MJXc-TeX-size3-R,MJXc-TeX-size3-Rw} .MJXc-TeX-size4-R {font-family: MJXc-TeX-size4-R,MJXc-TeX-size4-Rw} .MJXc-TeX-vec-R {font-family: MJXc-TeX-vec-R,MJXc-TeX-vec-Rw} .MJXc-TeX-vec-B {font-family: MJXc-TeX-vec-B,MJXc-TeX-vec-Bx,MJXc-TeX-vec-Bw} @font-face {font-family: MJXc-TeX-ams-R; src: local('MathJax_AMS'), local('MathJax_AMS-Regular')} @font-face {font-family: MJXc-TeX-ams-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_AMS-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_AMS-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_AMS-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-B; src: local('MathJax_Caligraphic Bold'), local('MathJax_Caligraphic-Bold')} @font-face {font-family: MJXc-TeX-cal-Bx; src: local('MathJax_Caligraphic'); font-weight: bold} @font-face {font-family: MJXc-TeX-cal-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-R; src: local('MathJax_Fraktur'), local('MathJax_Fraktur-Regular')} @font-face {font-family: MJXc-TeX-frak-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-frak-B; src: local('MathJax_Fraktur Bold'), local('MathJax_Fraktur-Bold')} @font-face {font-family: MJXc-TeX-frak-Bx; src: local('MathJax_Fraktur'); font-weight: bold} @font-face {font-family: MJXc-TeX-frak-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Fraktur-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Fraktur-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Fraktur-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-BI; src: local('MathJax_Math BoldItalic'), local('MathJax_Math-BoldItalic')} @font-face {font-family: MJXc-TeX-math-BIx; src: local('MathJax_Math'); font-weight: bold; font-style: italic} @font-face {font-family: MJXc-TeX-math-BIw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-BoldItalic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-BoldItalic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-BoldItalic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-R; src: local('MathJax_SansSerif'), local('MathJax_SansSerif-Regular')} @font-face {font-family: MJXc-TeX-sans-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-B; src: local('MathJax_SansSerif Bold'), local('MathJax_SansSerif-Bold')} @font-face {font-family: MJXc-TeX-sans-Bx; src: local('MathJax_SansSerif'); font-weight: bold} @font-face {font-family: MJXc-TeX-sans-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-sans-I; src: local('MathJax_SansSerif Italic'), local('MathJax_SansSerif-Italic')} @font-face {font-family: MJXc-TeX-sans-Ix; src: local('MathJax_SansSerif'); font-style: italic} @font-face {font-family: MJXc-TeX-sans-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_SansSerif-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_SansSerif-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_SansSerif-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-script-R; src: local('MathJax_Script'), local('MathJax_Script-Regular')} @font-face {font-family: MJXc-TeX-script-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Script-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Script-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Script-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-type-R; src: local('MathJax_Typewriter'), local('MathJax_Typewriter-Regular')} @font-face {font-family: MJXc-TeX-type-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Typewriter-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Typewriter-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Typewriter-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-cal-R; src: local('MathJax_Caligraphic'), local('MathJax_Caligraphic-Regular')} @font-face {font-family: MJXc-TeX-cal-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Caligraphic-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Caligraphic-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Caligraphic-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-B; src: local('MathJax_Main Bold'), local('MathJax_Main-Bold')} @font-face {font-family: MJXc-TeX-main-Bx; src: local('MathJax_Main'); font-weight: bold} @font-face {font-family: MJXc-TeX-main-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Bold.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-I; src: local('MathJax_Main Italic'), local('MathJax_Main-Italic')} @font-face {font-family: MJXc-TeX-main-Ix; src: local('MathJax_Main'); font-style: italic} @font-face {font-family: MJXc-TeX-main-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-main-R; src: local('MathJax_Main'), local('MathJax_Main-Regular')} @font-face {font-family: MJXc-TeX-main-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Main-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Main-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Main-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-math-I; src: local('MathJax_Math Italic'), local('MathJax_Math-Italic')} @font-face {font-family: MJXc-TeX-math-Ix; src: local('MathJax_Math'); font-style: italic} @font-face {font-family: MJXc-TeX-math-Iw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Math-Italic.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Math-Italic.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Math-Italic.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size1-R; src: local('MathJax_Size1'), local('MathJax_Size1-Regular')} @font-face {font-family: MJXc-TeX-size1-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size1-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size1-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size1-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size2-R; src: local('MathJax_Size2'), local('MathJax_Size2-Regular')} @font-face {font-family: MJXc-TeX-size2-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size2-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size2-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size2-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size3-R; src: local('MathJax_Size3'), local('MathJax_Size3-Regular')} @font-face {font-family: MJXc-TeX-size3-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size3-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size3-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size3-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-size4-R; src: local('MathJax_Size4'), local('MathJax_Size4-Regular')} @font-face {font-family: MJXc-TeX-size4-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Size4-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Size4-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Size4-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-R; src: local('MathJax_Vector'), local('MathJax_Vector-Regular')} @font-face {font-family: MJXc-TeX-vec-Rw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Regular.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Regular.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Regular.otf') format('opentype')} @font-face {font-family: MJXc-TeX-vec-B; src: local('MathJax_Vector Bold'), local('MathJax_Vector-Bold')} @font-face {font-family: MJXc-TeX-vec-Bx; src: local('MathJax_Vector'); font-weight: bold} @font-face {font-family: MJXc-TeX-vec-Bw; src /*1*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/eot/MathJax_Vector-Bold.eot'); src /*2*/: url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/woff/MathJax_Vector-Bold.woff') format('woff'), url('https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.2/fonts/HTML-CSS/TeX/otf/MathJax_Vector-Bold.otf') format('opentype')} cdc1,1−1,2d2,−10,0

In this particular example, cooperate corresponds to spending one of your own utility to give the other player two utility, and defect corresponds to doing nothing. This can represent a situation with the possibility of mutual benefit from cooperation, but where it is possible to win even more (at the other players expense) by cheating.

But we can also consider a negative game:


Here cooperate is doing nothing, while defect corresponds to gaining one utility for yourself while costing the other player two utility. This can represent burning the commons (if the players defect) or not (if they cooperate).

2. Explanation of Game of Chicken by comparing it to Prisoners Dilemma

Just like Prisoners Dilemma, Game of Chicken is a two player game, where each player can choose between two actions. These actions are typically called "swerve" and "straight", but in this blog post I will instead call the two actions "cooperate" and "defect" as to more easily compare with Prisoners Dilemma.

Also the same as Prisoners Dilemma: In Game of Chicken, I get the best payout if I defect and you cooperate (and vice versa). The difference is that conditional on you defecting, it is better for me if i cooperate.

A two action, two player game is a Game of Chicken if:

  • Holding my action constant it is better for me if you cooperate.
  • If you cooperate it is better for me if I defect.
  • If you defect is better for me to cooperate.
  • Cooperate-cooperate is Pareto optimal (even when including mixed strategies).

Furthermore, defect-defect is traditionally super bad for both players. But I would not say that this is a necessary condition for something to be a Game of Chicken.

Example payoff matrix:


The interesting part here is that I can pressure you to cooperate by credibly convincing you that I will defect. In other words, there is a first mover advantage, the first one to precommit to defecting will win against a rational player. However, this fact is of course known by every rational agent, so it might be a rational move to pre-commit to always defect in such games, no mater what. Then again, if two players with such commitments meet, they will both lose.

3. Blackmail is a Game of Chicken

I think that this is easiest explained by just writing out an example payout matrix

don't blackmailblackmailgive in (if blackmail)0,0−2,1don't give in (if blackmail)0,0−10,−10

If the the blackmailed player gives in, then they pay two utility to give the other player one utility. If the blackmailed player doesn't give in the blackmailer will carry out the threat which is costing both players ten utility. If the the blackmailer doesn't actually blackmails, than nothing happens.

Compare this to the example payout matrix of Game of Chicken. The blackmail payout matrix is not exactly the same, but I claim that in essence this is the same game. If you can handle Game of Chicken then you can handle blackmail both as the blackmailer and the blackmailed.

Not all blackmail is a Game of Chicken. If there is not cost in carrying out the threat then we are in a different type of situation. However I expect this to be rare. It seems unlikely to me that there is no opportunity cost at all in carrying out the threat. Further more, even if costless threats exists in some situations this does not invalidate the argument for considering those blackmail situation where there is a cost to the blackmailer to carryout the threat.

If the blackmailer gains utility by carrying out the threat then I would argue that it is not exactly blackmail anymore. If I have an action that I can take that would help me but hurt you and I ask you for some compensation for refraining from taking this action, then this is more like a value trade than a blackmail.

4. Why we should care about blackmail/Game of Chicken

Prisoners Dilemma receives a lot of attention because this class of games represents an important type of situation in most multiplayer environments. I claim that this is also true for Game of Chicken.

In any situation where one agent (A) has the ability to use up some of its own resources to impose a cost on another agent (B), then A can choose to blackmail B, thus creating a Game of Chicken like situation. And if A thinks that it can win this game, then it will be tempted to engage in blackmail.

If you expect that:

  • It is important to build AI's that can act well in multi-agent situations (e.g. because there will be several simultaneous AIs that are similarly powerful, or there will be acausal trade and threats between agents in different universes simulating each other)


  • Toy model such as Prisoners Dilemma are useful

then you should also care about Game of Chicken.

5. What to do? Iterated Game of Chicken?

What should we do about these insights? I am not sure yet. But one possible directions is to study iterated Game of Chicken.

Abram Demski argues that In Logical Time, All Games are Iterated Games. Basically if agents are simulating each other then this is sort of equivalent to the agent playing an iterated game.

Question for the comment section: What would be the winning strategy in iterated Game of Chicken?

I might run a tournament with different strategies.

This post was written with the support of the EA Hotel


Retrospective on a quantitative productivity logging attempt

22 марта, 2019 - 07:05
Published on March 22, 2019 2:31 AM UTC

I have a productivity scale I've used to log productivity data for several years. It's a subjective rating system from 1 to 10, and looks something like

1. can’t do anything, even reading. Worktime 0%.2. can force myself to read or work, but I can barely parse anything. Worktime 5%....5. I don’t want to work and am easily distracted, but I’m getting stuff done. 50%6. Some mild distractions, but I can stick to a pomodoro timer. Worktime 60%. ...10. Insane art-level hyperfocus. Worktime 100%.

At the end of each workday I would record how well I thought I'd done by this scale.

I'd been dissatisfied with this for a while – there was no way my brain was accurately tracking what percentage of time I was working, these descriptions are not well defined, don't cleanly map to some level of productivity. I can't prevent internal mapping drift, where my standards slowly rise or fall, such that a day I mark as productivity=4 this week is actually much more productive than a day I marked at 4 several years ago.

I'm invested in having good measurements, because I've been iterating on antidepressants and ADD meds for years, and having data on which ones are working on what metrics (I also track mood, exercise, sleep, social time) would be very useful for having a better life.

So I wrote a small Python script that tracked how much time I spent working at my job. Every time I took a break or went back to work, I'd mark it. If I noticed I'd started working during breaktime or zoning out during worktime, I'd 'revert' however many minutes I thought it had been to be of the other type. I also had 'dead' time where I wasn't getting anything done for reasons unrelated to my productivity, which I used to mark meetings or lunch breaks. At the end of the session, it would spit out a summary of how much time I'd spent working vs resting. This was a much more accurate, quantitative way of measuring what I wanted, or so I thought. I used it for a month and a week before I stopped.

Here's why I quit it.

  • As is frequently the case, using part of a system to monitor that system changes that system enough that the output of the monitoring is less useful. On bad attention days, I would be switching back and forth between work every two minutes. The work/break context switches were costlier, and seeing how short my work periods were lowered my mood.
  • The varying difficulty of tasks threw off the measurement. Some days I'd have monotonous but easy work, sometimes I'd have one tricky complicated thing that took a lot of brainpower and persistence. When I was using the subjective scale, I'd adjust my score for perceived task-easiness. But when I was using the time tracker, a day that was spent on "going through the codebase and adjusting every method call to do a new thing" would be logged as productive, even though I could have done that on that even on a bad day.
  • My personal productivity scale ranged between 0% and 100% productivity, and my ratings usually fall between 2 (subjective 5% work time) to 6 (60%). But my time-tracked work time usually hovered around 50%. I have an internal sense of "I haven't done enough work, I really need to do something" that kicks in and makes me do work-like activities badly, slowly, and ineffectively. For example, I'll read some documentation, staring at one sentence at a time, forcing myself to process it before moving onto the next one. That counts as work by the tracker time -- I'm certainly not resting -- and I can fill half my workday with that. At the end, the tracker will say I worked 50% of the time, but my subjective scale would say it was a 2. And I think my subjective scale is more correct.

I considered the idea of rating work periods every time I ended one, so that after spending an hour laboriously shoving sentences against my eyeballs, I'd indicate to the program that "taking break now; also, that last work period was only 10% of a real work period". But that goes right back to the problem of subjectively rated work, plus adds to the already-painful overhead I described in point 1.

That was an interesting lesson to learn. In the future, when I'm trying to measure something, I'll try to ask myself

  • How will integrating monitoring into my system change my system?
  • Describe a day that the proposed monitoring system would give a high score, but actually should have a bad score. How common do you expect these days to be?
  • Describe a day that would get a low score should have a high score, etc.
  • How much overhead do you expect this to add?


Declarative Mathematics

21 марта, 2019 - 22:05
Published on March 21, 2019 7:05 PM UTC

Programmers generally distinguish between “imperative” languages in which you specify what to do (e.g. C) versus “declarative” languages in which you specify what you want, and let the computer figure out how to do it (e.g. SQL). Over time, we generally expect programming to become more declarative, as more of the details are left to the compiler/interpreter. Good examples include the transition to automated memory management and, more recently, high-level tools for concurrent/parallel programming.

It’s hard to say what programming languages will look like in twenty or fifty years, but it’s a pretty safe bet that they’ll be a lot more declarative.

I expect that applied mathematics will also become much more declarative, for largely the same reasons: as computers grow in power and software expands its reach, there will be less and less need for (most) humans to worry about the details of rote computation.

What does this look like? Well, let’s start with a few examples of “imperative” mathematics:

  • Most grade-school arithmetic: it’s explicitly focused on computation, and even spells out the exact steps to follow (e.g. long division).
  • Gaussian reduction, as typically taught in a first-semester linear algebra class. It’s the undergrads’ version of grade-school arithmetic.
  • Most of the computation performed by hand in physics, engineering and upper-level econ courses & research, i.e. algebra/DEs/PDEs.

Contrast to the declarative counterparts:

  • Figure out what arithmetic needs to be done (i.e. what numbers to plug in) and then use a calculator
  • Set up a system of linear equations, then have python or wolfram invert the matrix
  • Choose which phenomena to include in a model, set up the governing equations, then use either numerical simulation (for pretty graphs) or a computer algebra system (for asymptotics and scaling relations).

In the declarative case, most of the work is in formulating the problem, figuring out what questions to ask, and translating it all into a language which a computer can work with - numbers, or matrices, or systems of equations.

This is all pretty standard commentary at the level of mathematics education, but the real importance is in shaping the goals of applied mathematics. For the past century, the main objectives of mathematical research programs would be things like existence & uniqueness, or exhaustive classification of some objects, or algorithms for solving some problem (a.k.a. constructive solution/proof). With the shift toward declarative mathematics, there will be more focus on building declarative frameworks for solving various kinds of problems.

The best example I know of is convex analysis, in the style taught by Stephen Boyd (course, book). Boyd’s presentation is the user’s guide to convex optimization: it addresses what kinds of questions can be asked/answered, how to recognize relevant applications in the wild, how to formulate problems, what guarantees are offered in terms of solutions, and of course a firehose of examples from a wide variety of fields. In short, it includes exactly the pieces needed to use the tools of convex analysis as a declarative framework. By contrast, the internals of optimization algorithms are examined only briefly, with little depth and a focus on things which a user might need to tweak. Complicated proofs are generally omitted altogether, the relevant results simply stated as tools available for use.

This is what a mature declarative mathematical framework looks like: it provides a set of tools for practitioners to employ on practical problems. Users don’t need to know what’s going on under the hood; the algorithms and proofs generally “just work” without the user needing to worry about the details. The user’s job is to understand the language of the framework, the interface, and translate their own problems into that language. Once they’ve expressed what they want, the tools take over and handle the rest.

That’s the big goal of future mathematical disciplines: provide a practical framework which practitioners can use to solve real-world problems in the wild, without having to know all the little details and gotchas under the hood.

One last example, which is particularly relevant to me and to ML/AI research. One of the overarching goals of probability/statistics/ML is to be able to code up a generative model, pass it into a magical algorithm, and get back parameter estimates and uncertainties. The “language” of generative models is very intuitive and generally easy to work with, making it an excellent interface to a declarative mathematical toolkit. Unfortunately, the behind-the-scenes part of the toolkit remains relatively finicky and inefficient. As of today, the “magical algorithm” part is usually MCMC, which is great in terms of universality but often super-exponentially slow for multimodal problems, especially in high dimensions, and can converge very slowly even in simple unimodal problems. It’s not really reliable enough to use without thinking about what’s under the hood. Better mathematical tools and guarantees are needed before this particular framework fully matures.

If anyone has other examples of maturing or up-and-coming declarative mathematical frameworks, I’d be very interested to hear about them.


The Main Sources of AI Risk?

21 марта, 2019 - 21:28
Published on March 21, 2019 6:28 PM UTC

There are so many causes or sources of AI risk that it's getting hard to keep them all in mind. I propose we keep a list of the main sources (that we know about), such that we can say that if none of these things happen, then we've mostly eliminated AI risk (as an existential risk) at least as far as we can determine. Here's a list that I spent a couple of hours enumerating and writing down. Did I miss anything important?

  1. Insufficient time/resources for AI safety (for example caused by intelligence explosion or AI race)
  2. Insufficient global coordination, leading to the above
  3. Misspecified or incorrectly learned goals/values
  4. Inner optimizers
  5. ML differentially accelerating easy to measure goals
  6. Paul's "influence-seeking behavior" (a combination of 3 and 4 above?)
  7. AI generally accelerating intellectual progress in a wrong direction (e.g., accelerating unsafe/risky technologies more than knowledge/wisdom about how to safely use those technologies)
  8. Metaethical error
  9. Metaphilosophical error
  10. Other kinds of philosophical errors in AI design (e.g., giving AI a wrong prior or decision theory)
  11. Other design/coding errors (e.g., accidentally putting a minus sign in front of utility function, supposedly corrigible AI not actually being corrigible)
  12. Doing acausal reasoning in a wrong way (e.g., failing to make good acausal trades, being acausally extorted, failing to acausally influence others who can be so influenced)
  13. Human-controlled AIs ending up with wrong values due to insufficient "metaphilosophical paternalism"
  14. Human-controlled AIs causing ethical disasters (e.g., large scale suffering that can't be "balanced out" later) prior to reaching moral/philosophical maturity
  15. Intentional corruption of human values
  16. Unintentional corruption of human values
  17. Mind crime (disvalue unintentionally incurred through morally relevant simulations in AIs' minds)
  18. Premature value lock-in (i.e., freezing one's current conception of what's good into a utility function)
  19. Extortion between AIs leading to vast disvalue
  20. Distributional shifts causing apparently safe/aligned AIs to stop being safe/aligned
  21. Value drift and other kinds of error as AIs self-modify, or AIs failing to solve value alignment for more advanced AIs
  22. Treacherous turn / loss of property rights due to insufficient competitiveness of humans & human-aligned AIs
  23. Gradual loss of influence due to insufficient competitiveness of humans & human-aligned AIs
  24. Utility maximizers / goal-directed AIs having an economic and/or military competitive advantage due to relative ease of cooperation/coordination, defense against value corruption and other forms of manipulation and attack, leading to one or more of the above
  25. In general, the most competitive type of AI being too hard to align or to safely use
  26. Computational resources being too cheap, leading to one or more of the above

(With this post I mean to (among other things) re-emphasize the disjunctive nature of AI risk, but this list isn't fully disjunctive (i.e., some of the items are subcategories or causes of others), and I mostly gave a source of AI risk its own number in the list if it seemed important to make that source more salient. Maybe once we have a list of everything that is important, it would make sense to create a graph out of it.)


[Link] IDA 9/14: The Scheme

21 марта, 2019 - 21:28
Published on March 21, 2019 6:28 PM UTC

This is a linkpost for https://app.grasple.com/#/level/1669

Every Thursday for 4 weeks, we will be posting lessons about Iterated Distillation and Amplification. They're largely based on Paul Christiano's sequence here on LW. He graciously allowed us to use his work.

This is the third section, containing one lesson synthesizing the blog posts detailing the scheme and the first of our video series on the scheme, more of which will be coming in the next few weeks. The video can also be found on Youtube .

Note that access to the lessons requires creating an account here.

Have a nice day!


What should we expect from GPT-3?

21 марта, 2019 - 17:28
Published on March 21, 2019 2:28 PM UTC

When it will appear? (My guess is 2020).

Will it be created by OpenAI and will it be advertised? (My guess is that it will not be publicly known until 2021, but other companies may create open versions before it.)

How much data will be used for its training and what type of data? (My guess is 400 GB of text plus illustrating pictures, but not audio and video.)

What it will be able to do? (My guess: translation, picture generation based on text, text generation based on pictures – with 70 per cent of human performance.)

How many parameters will be in the model? (My guess is 100 billion to trillion.)

How much compute will be used for training? (No idea.)


[Question] Tracking accuracy of personal forecasts

20 марта, 2019 - 23:40
Published on March 20, 2019 8:39 PM UTC

I've been thinking how I can improve my accuracy predicting events of personal interest (e.g., "Will my landlord get the washing machine fixed within the next two weeks", or "Will my parent die this year" for a more extreme example). Betting markets will not help me with that.

At first I thought about creating dedicated software that gathers such predictions, the final outcomes of predicted events, and presents their accuracy so that the user can spot bias. Then I realised a simple spreadsheet might suffice to gather data at first and assess how useful this is. And if the need arises in the future, it should be easy to import into dedicated software, provided that all the relevant data is already there.

Does anyone track their personal predictions? If so, what methodology do you use, and did it allow you to improve your accuracy?

As an RFC, here's the spreadsheet layout I have on mind:

  • Tags: (value 0 or 1):
    • Health
    • Finance
    • Interpersonal relations
    • ...
  • Date of the forecast
  • Event (e.g., "My landlord will get the washing machine fixed within the next two weeks"). I'm planning to formulate them so that "yes" is always the desired outcome, so that it's easy to spot if I'm reliably too optimistic or pessimistic.
  • Estimated probability
  • Deadline of the forecast
  • Outcome (value 0 or 1, filled after the deadline of the forecast, or when the answer is known sooner)


Does criticism catalyze analytical thinking in groups?

20 марта, 2019 - 19:27
Published on March 20, 2019 4:27 PM UTC

Brainstorming, a technique long thought to enhance group creativity. Although the research shows little evidence of effectiveness, this technique (Alex Faickney Osborn, 1957) has persisted in practice. More specifically, brainstorming instructions increase the number of ideas in a group but are usually less than the total number of ideas generated by the same number of brainstorming individuals alone (Brown and Paulus, 1996).

The instructions for brainstorming are fairly precise:

  • Quantity: come up with as many ideas as you can.
  • Do not criticise others’ ideas.
  • Build on others’ ideas.
  • Freewheeling is welcome.

Not to criticise each others’ idea is an admonition as it may cause evaluation apprehension: people will be reluctant to express creative ideas; they won’t free wheel for fear of evaluation and risk.

We’re traditionally taught that the criticism inherent to analysis is a no-go in the process of ideation: the second rule of brainstorming is don’t criticise. The absence of criticism is supposed to set free creativity but as it happens, the human mind is not a creative volcano waiting for an excuse to erupt.

Charlan Nemeth, a professor of psychology at the University of California, demonstrated that people are not very creative when they are given complete freedom. As part of an experiment on word association, she asked people to associate freely on the word blue, the vast majority said first green, then sky or ocean. Her research shows that people are able to generate both a larger number of associations and more creative associations when faced with dissent and criticism.

As part of the aforementioned experiment, Nemeth placed her test subjects alongside actors posing as test subjects who would disagree on established truths like what colour an object is. It turned out that the test subjects confronted with the dissenting actors had much more creative associations than the test subjects in the group without actors. Rather than associating blue with green, sky, or ocean, they associated blue with words like berry pie. Instead of killing creativity, disagreement seems to foster it.

In dire straits, analysis becomes an essential tool to understand what is going on in the world and help evaluate existing ideas—it is also a catalyst of the creativity needed to come up with solutions.

Analysis injects dissent and criticism into the ideation process, and contrary to what the rules of brainstorming tell us, this makes people more, not less, creative.


Games in Kocherga club: Fallacymania, Tower of Chaos, Scientific Discovery

20 марта, 2019 - 16:52
Published on March 20, 2019 1:52 PM UTC

Welcome to Moscow LW community makeshift games! In that games, some rationality skills are involved, so you can practise while you playing!
* Fallacymania: it is a game where you guess logical fallacies in arguments, or practise using logical fallacies yourself (depending on team in which you will be).
* Tower of Chaos: funny game with guessing the rules of human placement on a Twister mat.
* Scientific Discovery: modified version of Zendo with simultaneous turns for all players.
Details about the games: https://bit.ly/2J2T5o8
Come to antikafe "Kocherga", ul.B.Dorogomilovskaya, 5-2. The map is here: https://kocherga-club.ru/#contacts
Games begin at 19:40, the length is 3 hours.


Moscow LW meetup in "Nauchka" library

20 марта, 2019 - 16:49
Published on March 20, 2019 1:49 PM UTC

Welcome to the next Moscow LW meetup in "Nauchka" library!

Our plan:
* Street Epistemology: theory and practice.
* Fallacymania game.
* Roleplaying game "Technoquest".

Details about Fallacymania can be found here: https://bit.ly/2J2T5o8
Meetup details are here: https://www.facebook.com/events/2096340610448015
Come to "Nauchka", ul.Dubininskaya, 20. Nearest metro station is Paveletskaya. Map is here: http://nauchka.ru/contacts/
Meetup begins at 14:00, the length is 6 hours.


What's wrong with these analogies for understanding Informed Oversight and IDA?

20 марта, 2019 - 12:11
Published on March 20, 2019 9:11 AM UTC

In Can HCH epistemically dominate Ramanujan? Alex Zhu wrote:

If HCH is ascription universal, then it should be able to epistemically dominate an AI theorem-prover that reasons similarly to how Ramanujan reasoned. But I don’t currently have any intuitions as to why explicit verbal breakdowns of reasoning should be able to replicate the intuitions that generated Ramanujan’s results (or any style of reasoning employed by any mathematician since Ramanujan, for that matter).

And I answered:

My guess is that HCH has to reverse engineer the theorem prover, figure out how/why it works, and then reproduce the same kind of reasoning.

And then I followed up my own comment with:

It occurs to me that if the overseer understands everything that the ML model (that it’s training) is doing, and the training is via some kind of local optimization algorithm like gradient descent, the overseer is essentially manually programming the ML model by gradually nudging it from some initial (e.g., random) point in configuration space.

No one answered my comments with either a confirmation or denial, as to whether these guesses of how to understand Universality / Informed Oversight and IDA are correct. I'm surfacing this question as a top-level post because if "Informed Oversight = reverse engineering" and "IDA = programming by nudging" are good analogies for understanding Informed Oversight and IDA, it seems to have pretty significant implications.

In particular it seems to imply that there's not much hope for IDA to be competitive with ML-in-general, because if IDA is analogous to a highly constrained method of "manual" programming, that seems unlikely to be competitive with less constrained methods of "manual" programming (i.e., AIs designing and programming more advanced AIs in more general ways, similar to how humans do most programming today), which itself is presumably not competitive with general (unconstrained-by-safety) ML (otherwise ML would not be the competitive benchmark).

If these are not good ways to understand IO and IDA, can someone please point out why?